2023-12-13T23:43:49.981 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-13T23:43:49.981 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} --verbose --archive /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669 --owner scheduled_yuriw@teuthology --name yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/orig.config.yaml 2023-12-13T23:43:50.025 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-13T23:43:50.111 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669 branch: wip-yuri10-testing-2023-12-12-1229 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} email: yweinste@redhat.com first_in_suite: false job_id: '7490669' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri10-testing-2023-12-12-1229 ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri10-testing-2023-12-12-1229 sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 8548 sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri10-testing-2023-12-12-1229 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi124.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBqrEjhIl+df54aDgFucXunpO4UgObFWJ9ytkz5DhJ26N/0p3E8peMvYr//WC8rqdADbgVMPgTdeyRMyUl8nI8= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJA14Vswc23fItM+D5R1LMCctN4imLdwSpzC1urUzN+ZX/tklb8W2t0fbPe7MwqXCylSmjMNwo57jeZjMMTgPO0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-13_20:38:28 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-13T23:43:50.112 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa; will attempt to use it 2023-12-13T23:43:50.112 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks 2023-12-13T23:43:50.113 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-13T23:43:50.114 INFO:teuthology.task.internal:Checking packages... 2023-12-13T23:43:50.141 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '021ac1670a21c19759080c6da90baaeb42e7d175' 2023-12-13T23:43:50.141 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-13T23:43:50.141 INFO:teuthology.packaging:ref: None 2023-12-13T23:43:50.141 INFO:teuthology.packaging:tag: None 2023-12-13T23:43:50.141 INFO:teuthology.packaging:branch: wip-yuri10-testing-2023-12-12-1229 2023-12-13T23:43:50.141 INFO:teuthology.packaging:sha1: 021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-13T23:43:50.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri10-testing-2023-12-12-1229 2023-12-13T23:43:50.352 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-157.g021ac167 2023-12-13T23:43:50.354 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-13T23:43:50.360 INFO:teuthology.task.internal:no buildpackages task found 2023-12-13T23:43:50.360 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-13T23:43:50.407 INFO:teuthology.task.internal:Saving configuration 2023-12-13T23:43:50.425 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-13T23:43:50.434 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-13T23:43:50.460 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-13 23:38:45.291895', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBqrEjhIl+df54aDgFucXunpO4UgObFWJ9ytkz5DhJ26N/0p3E8peMvYr//WC8rqdADbgVMPgTdeyRMyUl8nI8='} 2023-12-13T23:43:50.486 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-13 23:38:45.293484', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJA14Vswc23fItM+D5R1LMCctN4imLdwSpzC1urUzN+ZX/tklb8W2t0fbPe7MwqXCylSmjMNwo57jeZjMMTgPO0='} 2023-12-13T23:43:50.486 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-13T23:43:50.492 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-12-13T23:43:50.493 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-12-13T23:43:50.493 INFO:teuthology.run_tasks:Running task console_log... 2023-12-13T23:43:50.634 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2557302e50>, signals=[15]) 2023-12-13T23:43:50.635 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-13T23:43:50.640 INFO:teuthology.task.internal:Opening connections... 2023-12-13T23:43:50.641 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:43:50.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:43:50.769 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:43:50.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:43:50.853 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-13T23:43:50.860 DEBUG:teuthology.orchestra.run.smithi124:> uname -m 2023-12-13T23:43:50.877 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2023-12-13T23:43:50.877 DEBUG:teuthology.orchestra.run.smithi124:> cat /etc/os-release 2023-12-13T23:43:50.932 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="CentOS Stream" 2023-12-13T23:43:50.932 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="9" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:ID="centos" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="rhel fedora" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="9" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el9" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:LOGO="fedora-logo-icon" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://centos.org/" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-13T23:43:50.933 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-13T23:43:50.934 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-13T23:43:50.934 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2023-12-13T23:43:50.959 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2023-12-13T23:43:50.978 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2023-12-13T23:43:50.979 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2023-12-13T23:43:51.035 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-13T23:43:51.036 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-13T23:43:51.037 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2023-12-13T23:43:51.067 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-13T23:43:51.076 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-13T23:43:51.083 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-13T23:43:51.083 DEBUG:teuthology.orchestra.run.smithi124:> test '!' -e /home/ubuntu/cephtest 2023-12-13T23:43:51.086 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2023-12-13T23:43:51.103 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-13T23:43:51.109 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-13T23:43:51.110 DEBUG:teuthology.orchestra.run.smithi124:> test -z $(ls -A /var/lib/ceph) 2023-12-13T23:43:51.143 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2023-12-13T23:43:51.258 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-13T23:43:51.314 INFO:teuthology.run_tasks:Running task kernel... 2023-12-13T23:43:51.336 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-13T23:43:51.336 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-13T23:43:51.337 DEBUG:teuthology.orchestra.run.smithi124:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-13T23:43:51.337 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-13T23:43:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:43:51.353 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2023-12-13T23:43:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:43:51.354 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2023-12-13T23:43:51.411 INFO:teuthology.orchestra.run.smithi124.stdout:5.14.0-361.el9.x86_64 2023-12-13T23:43:51.411 INFO:teuthology.task.kernel:Running kernel on smithi124: 5.14.0-361.el9.x86_64 2023-12-13T23:43:51.411 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum install -y kernel 2023-12-13T23:43:51.412 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2023-12-13T23:43:51.412 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2023-12-13T23:43:51.412 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2023-12-13T23:43:53.597 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS-9 - AppStream 11 MB/s | 6.4 MB 00:00 2023-12-13T23:43:53.614 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 14 MB/s | 6.4 MB 00:00 2023-12-13T23:43:54.930 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2023-12-13T23:43:54.947 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 7.7 MB/s | 2.0 MB 00:00 2023-12-13T23:43:55.738 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 7.9 MB 00:00 2023-12-13T23:43:56.624 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 7.9 MB 00:01 2023-12-13T23:43:58.581 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - AppStream 10 MB/s | 18 MB 00:01 2023-12-13T23:43:59.431 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 11 MB/s | 18 MB 00:01 2023-12-13T23:44:03.123 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - CRB 4.7 MB/s | 5.8 MB 00:01 2023-12-13T23:44:04.616 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2023-12-13T23:44:04.757 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.8 MB 00:01 2023-12-13T23:44:06.191 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2023-12-13T23:44:08.095 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 11 MB/s | 20 MB 00:01 2023-12-13T23:44:10.584 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 3.4 MB/s | 20 MB 00:05 2023-12-13T23:44:12.778 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2023-12-13T23:44:14.842 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-13T23:44:14.843 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-13T23:44:14.941 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-12-13T23:44:14.952 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2023-12-13T23:44:14.953 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2023-12-13T23:44:14.954 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:44:14.954 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 96 M 2023-12-13T23:44:14.954 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 121 M 2023-12-13T23:44:14.954 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-12-13T23:44:15.242 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2023-12-13T23:44:15.712 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 8.8 MB/s | 5.6 MB 00:00 2023-12-13T23:44:16.488 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 14 MB/s | 20 MB 00:01 2023-12-13T23:44:17.349 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-13T23:44:17.349 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-13T23:44:17.388 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 16 MB/s | 38 MB 00:02 2023-12-13T23:44:17.454 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout: Package Architecture Version Repository Size 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout: kernel x86_64 5.14.0-391.el9 baseos 5.6 M 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-12-13T23:44:17.463 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-core x86_64 5.14.0-391.el9 baseos 20 M 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-modules x86_64 5.14.0-391.el9 baseos 38 M 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-modules-core x86_64 5.14.0-391.el9 baseos 32 M 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout:================================================================================ 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout:Install 4 Packages 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:44:17.464 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 96 M 2023-12-13T23:44:17.465 INFO:teuthology.orchestra.run.smithi124.stdout:Installed size: 121 M 2023-12-13T23:44:17.465 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-12-13T23:44:17.986 INFO:teuthology.orchestra.run.smithi124.stdout:(1/4): kernel-5.14.0-391.el9.x86_64.rpm 12 MB/s | 5.6 MB 00:00 2023-12-13T23:44:18.436 INFO:teuthology.orchestra.run.smithi124.stdout:(2/4): kernel-core-5.14.0-391.el9.x86_64.rpm 22 MB/s | 20 MB 00:00 2023-12-13T23:44:18.763 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 11 MB/s | 32 MB 00:03 2023-12-13T23:44:18.763 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-12-13T23:44:18.763 INFO:teuthology.orchestra.run.smithi203.stdout:Total 25 MB/s | 96 MB 00:03 2023-12-13T23:44:18.986 INFO:teuthology.orchestra.run.smithi124.stdout:(3/4): kernel-modules-5.14.0-391.el9.x86_64.rpm 26 MB/s | 38 MB 00:01 2023-12-13T23:44:19.486 INFO:teuthology.orchestra.run.smithi124.stdout:(4/4): kernel-modules-core-5.14.0-391.el9.x86_6 22 MB/s | 32 MB 00:01 2023-12-13T23:44:19.487 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-12-13T23:44:19.487 INFO:teuthology.orchestra.run.smithi124.stdout:Total 48 MB/s | 96 MB 00:02 2023-12-13T23:44:19.617 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-12-13T23:44:20.040 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-12-13T23:44:20.040 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-12-13T23:44:20.342 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-12-13T23:44:20.726 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-12-13T23:44:20.726 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-12-13T23:44:21.343 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-12-13T23:44:21.344 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-12-13T23:44:22.024 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-12-13T23:44:22.024 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-12-13T23:44:24.211 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-12-13T23:44:24.647 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-12-13T23:44:24.923 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-13T23:44:25.048 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:44:25.347 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : kernel-modules-core-5.14.0-391.el9.x86_64 1/4 2023-12-13T23:44:25.481 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:44:27.828 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:44:28.062 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:44:28.246 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:44:28.431 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:44:32.655 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:44:32.786 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:44:33.038 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:44:33.156 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : kernel-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:44:37.404 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:44:37.777 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.372 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.415 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.530 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.571 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-modules-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.844 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.844 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-13T23:45:06.844 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:45:06.845 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:45:06.959 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: kernel-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:06.959 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : kernel-5.14.0-391.el9.x86_64 1/4 2023-12-13T23:45:06.960 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : kernel-core-5.14.0-391.el9.x86_64 2/4 2023-12-13T23:45:06.960 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : kernel-modules-5.14.0-391.el9.x86_64 3/4 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.283 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.284 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:45:07.284 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-12-13T23:45:07.370 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : kernel-modules-core-5.14.0-391.el9.x86_64 4/4 2023-12-13T23:45:07.370 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:45:07.370 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-12-13T23:45:07.370 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.371 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-core-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.371 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-modules-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.371 INFO:teuthology.orchestra.run.smithi124.stdout: kernel-modules-core-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.371 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:45:07.371 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-12-13T23:45:07.792 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2023-12-13T23:45:07.919 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2023-12-13T23:45:07.937 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:07.937 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-13T23:45:07.937 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-13T23:45:07.937 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-13T23:45:07.938 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-13T23:45:07.938 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-13T23:45:07.938 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2023-12-13T23:45:07.938 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2023-12-13T23:45:08.055 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:08.055 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.055 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2023-12-13T23:45:08.059 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:08.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-13T23:45:08.059 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-13T23:45:08.059 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-13T23:45:08.059 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi124.front.sepia.ceph.com, path=None, version=distro) 2023-12-13T23:45:08.059 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2023-12-13T23:45:08.097 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2023-12-13T23:45:08.097 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2023-12-13T23:45:08.097 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2023-12-13T23:45:08.098 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:45:08.099 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2023-12-13T23:45:08.100 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-13T23:45:08.100 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.100 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-13T23:45:08.176 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-5.14.0-391.el9.x86_64 2023-12-13T23:45:08.177 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.177 DEBUG:teuthology.orchestra.run.smithi124:> sudo rpm -qi grub2-tools 2023-12-13T23:45:08.215 INFO:teuthology.orchestra.run.smithi124.stdout:Name : grub2-tools 2023-12-13T23:45:08.215 INFO:teuthology.orchestra.run.smithi124.stdout:Epoch : 1 2023-12-13T23:45:08.215 INFO:teuthology.orchestra.run.smithi124.stdout:Version : 2.06 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Release : 68.el9 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Architecture: x86_64 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Group : Unspecified 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Size : 8264254 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:License : GPLv3+ 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Packager : builder@centos.org 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:Vendor : CentOS 2023-12-13T23:45:08.216 INFO:teuthology.orchestra.run.smithi124.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:Summary : Support tools for GRUB. 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:Description : 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:hardware devices. 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:45:08.217 INFO:teuthology.orchestra.run.smithi124.stdout:This subpackage provides tools for support of all platforms. 2023-12-13T23:45:08.218 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-13T23:45:08.218 INFO:teuthology.task.kernel:Updating grub on smithi124 to boot 5.14.0-391.el9.x86_64 2023-12-13T23:45:08.218 DEBUG:teuthology.orchestra.run.smithi124:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-13T23:45:08.743 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2023-12-13T23:45:08.859 INFO:teuthology.orchestra.run.smithi124.stderr:Generating grub configuration file ... 2023-12-13T23:45:10.135 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-13T23:45:10.172 INFO:teuthology.orchestra.run.smithi203.stderr:done 2023-12-13T23:45:10.175 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2023-12-13T23:45:10.190 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.th2OlRx4lf 2023-12-13T23:45:10.190 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.th2OlRx4lf 2023-12-13T23:45:10.220 INFO:teuthology.orchestra.run.smithi124.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-13T23:45:10.251 INFO:teuthology.orchestra.run.smithi124.stderr:done 2023-12-13T23:45:10.253 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-12-13T23:45:10.258 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.th2OlRx4lf 2023-12-13T23:45:10.268 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.42ncdLv38E 2023-12-13T23:45:10.268 DEBUG:teuthology.orchestra.run.smithi124:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.42ncdLv38E 2023-12-13T23:45:10.337 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.42ncdLv38E 2023-12-13T23:45:10.394 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.th2OlRx4lf is 6KB 2023-12-13T23:45:10.443 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.th2OlRx4lf 2023-12-13T23:45:10.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2023-12-13T23:45:10.475 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.42ncdLv38E is 6KB 2023-12-13T23:45:10.522 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-13T23:45:10.522 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-13T23:45:10.522 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-13T23:45:10.522 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-13T23:45:10.523 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-13T23:45:10.525 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.42ncdLv38E 2023-12-13T23:45:10.538 DEBUG:teuthology.orchestra.run.smithi124:> sudo /bin/ls /boot/loader/entries || true 2023-12-13T23:45:10.603 INFO:teuthology.orchestra.run.smithi124.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-13T23:45:10.603 INFO:teuthology.orchestra.run.smithi124.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-13T23:45:10.603 INFO:teuthology.orchestra.run.smithi124.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-13T23:45:10.603 INFO:teuthology.orchestra.run.smithi124.stdout:15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64.conf 2023-12-13T23:45:10.605 DEBUG:teuthology.orchestra.run.smithi124:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-391.el9.x86_64 2023-12-13T23:45:10.663 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2023-12-13T23:45:10.761 DEBUG:teuthology.orchestra.run.smithi124:> sudo shutdown -r now 2023-12-13T23:45:40.695 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-13T23:45:40.696 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-12-13T23:45:40.697 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:45:40.767 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-13T23:45:40.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-12-13T23:45:40.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:46:15.186 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2023-12-13T23:46:24.196 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-12-13T23:46:24.197 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:46:40.785 DEBUG:teuthology.orchestra.remote:timed out 2023-12-13T23:46:49.794 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-12-13T23:46:49.794 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:46:50.212 DEBUG:teuthology.orchestra.run.smithi124:> true 2023-12-13T23:46:50.576 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-12-13T23:46:50.576 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-391.el9.x86_64"... 2023-12-13T23:46:50.576 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2023-12-13T23:46:50.590 INFO:teuthology.orchestra.run.smithi124.stdout:5.14.0-391.el9.x86_64 2023-12-13T23:46:50.590 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-13T23:46:50.590 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-13T23:46:50.591 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-13T23:46:51.592 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-13T23:46:51.592 DEBUG:teuthology.orchestra.run.smithi124:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-13T23:46:51.669 INFO:teuthology.orchestra.run.smithi124.stdout:ttyS1 2023-12-13T23:46:51.709 DEBUG:teuthology.parallel:result is None 2023-12-13T23:46:56.017 DEBUG:teuthology.orchestra.run.smithi203:> true 2023-12-13T23:46:56.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-12-13T23:46:56.421 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-391.el9.x86_64"... 2023-12-13T23:46:56.421 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2023-12-13T23:46:56.435 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-391.el9.x86_64 2023-12-13T23:46:56.435 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-391.el9.x86_64 vs 5.14.0-391.el9.x86_64 2023-12-13T23:46:56.435 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-13T23:46:56.435 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-13T23:46:57.436 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-13T23:46:57.436 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-13T23:46:57.510 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2023-12-13T23:46:57.541 DEBUG:teuthology.parallel:result is None 2023-12-13T23:46:57.541 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-13T23:46:57.549 INFO:teuthology.task.internal:Creating test directory... 2023-12-13T23:46:57.550 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-13T23:46:57.553 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-13T23:46:57.568 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-13T23:46:57.575 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-13T23:46:57.582 INFO:teuthology.task.internal:Creating archive directory... 2023-12-13T23:46:57.582 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-13T23:46:57.611 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-13T23:46:57.714 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-13T23:46:57.720 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-13T23:46:57.720 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-13T23:46:57.723 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-13T23:46:57.754 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-13T23:46:57.768 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-13T23:46:57.775 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-13T23:46:57.788 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-13T23:46:57.790 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-13T23:46:57.797 INFO:teuthology.task.internal:Configuring sudo... 2023-12-13T23:46:57.797 DEBUG:teuthology.orchestra.run.smithi124:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-13T23:46:57.812 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-13T23:46:57.842 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-13T23:46:57.851 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-13T23:46:57.852 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-13T23:46:57.885 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-13T23:46:57.899 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-13T23:46:57.987 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-13T23:46:58.038 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-13T23:46:58.123 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-13T23:46:58.169 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:46:58.169 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-13T23:46:58.238 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-13T23:46:58.283 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-13T23:46:58.332 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-13T23:46:58.413 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-13T23:46:58.457 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:46:58.457 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-13T23:46:58.524 DEBUG:teuthology.orchestra.run.smithi124:> sudo service rsyslog restart 2023-12-13T23:46:58.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2023-12-13T23:46:58.579 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-13T23:46:58.617 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-13T23:46:59.079 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-13T23:46:59.088 INFO:teuthology.task.internal:Starting timer... 2023-12-13T23:46:59.088 INFO:teuthology.run_tasks:Running task pcp... 2023-12-13T23:46:59.130 INFO:teuthology.run_tasks:Running task selinux... 2023-12-13T23:46:59.138 DEBUG:teuthology.orchestra.run.smithi124:> sudo service auditd rotate 2023-12-13T23:46:59.198 INFO:teuthology.orchestra.run.smithi124.stdout:Rotating logs: 2023-12-13T23:46:59.200 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2023-12-13T23:46:59.256 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2023-12-13T23:46:59.258 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-13T23:46:59.258 DEBUG:teuthology.orchestra.run.smithi124:> /usr/sbin/getenforce 2023-12-13T23:46:59.292 INFO:teuthology.orchestra.run.smithi124.stdout:Permissive 2023-12-13T23:46:59.293 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2023-12-13T23:46:59.313 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2023-12-13T23:46:59.314 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi124.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2023-12-13T23:46:59.314 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-13T23:46:59.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:46:59.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-13T23:46:59.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:46:59.391 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-13T23:46:59.391 DEBUG:teuthology.orchestra.run.smithi124:> sudo /usr/sbin/setenforce permissive 2023-12-13T23:46:59.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2023-12-13T23:46:59.473 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-13T23:46:59.483 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-13T23:46:59.483 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-13T23:46:59.500 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-13T23:46:59.501 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi124.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-13T23:52:37.114 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi124.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2023-12-13T23:52:37.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-12-13T23:52:37.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:52:37.203 DEBUG:teuthology.orchestra.run.smithi124:> true 2023-12-13T23:52:37.272 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-12-13T23:52:37.272 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-12-13T23:52:37.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-13T23:52:37.355 DEBUG:teuthology.orchestra.run.smithi203:> true 2023-12-13T23:52:37.428 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2023-12-13T23:52:37.428 INFO:teuthology.run_tasks:Running task clock... 2023-12-13T23:52:37.439 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-13T23:52:37.440 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-13T23:52:37.440 DEBUG:teuthology.orchestra.run.smithi124:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-13T23:52:37.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-13T23:52:37.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-13T23:52:37.471 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-13T23:52:37.487 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-13T23:52:37.500 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-13T23:52:37.516 INFO:teuthology.orchestra.run.smithi124.stderr:sudo: ntpd: command not found 2023-12-13T23:52:37.516 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-13T23:52:37.529 INFO:teuthology.orchestra.run.smithi124.stdout:506 Cannot talk to daemon 2023-12-13T23:52:37.546 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2023-12-13T23:52:37.546 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-13T23:52:37.559 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2023-12-13T23:52:37.561 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-13T23:52:37.575 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-13T23:52:37.591 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-13T23:52:37.604 INFO:teuthology.orchestra.run.smithi124.stderr:bash: line 1: ntpq: command not found 2023-12-13T23:52:37.607 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-13T23:52:37.607 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-12-13T23:52:37.608 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.608 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.608 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.608 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.634 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2023-12-13T23:52:37.638 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-13T23:52:37.638 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2023-12-13T23:52:37.638 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.639 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.639 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.639 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:52:37.639 INFO:teuthology.run_tasks:Running task install... 2023-12-13T23:52:37.647 DEBUG:teuthology.task.install:project ceph 2023-12-13T23:52:37.647 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}} 2023-12-13T23:52:37.648 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'} 2023-12-13T23:52:37.648 INFO:teuthology.task.install:Using flavor: default 2023-12-13T23:52:37.653 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-13T23:52:37.653 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2023-12-13T23:52:37.654 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'tag': None, 'wait_for_package': False} 2023-12-13T23:52:37.654 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-13T23:52:37.656 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'tag': None, 'wait_for_package': False} 2023-12-13T23:52:37.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-13T23:52:37.800 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/ 2023-12-13T23:52:37.801 INFO:teuthology.task.install.rpm:Package version is 19.0.0-157.g021ac167 2023-12-13T23:52:37.877 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/ 2023-12-13T23:52:37.877 INFO:teuthology.task.install.rpm:Package version is 19.0.0-157.g021ac167 2023-12-13T23:52:37.978 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-13T23:52:37.978 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:52:37.978 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-13T23:52:38.007 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-12-13T23:52:38.007 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/021ac1670a21c19759080c6da90baaeb42e7d175/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-13T23:52:38.050 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri10-testing-2023-12-12-1229/021ac1670a21c19759080c6da90baaeb42e7d175/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-13T23:52:38.050 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:52:38.050 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-13T23:52:38.075 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-13T23:52:38.079 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-12-13T23:52:38.079 DEBUG:teuthology.orchestra.run.smithi124:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/021ac1670a21c19759080c6da90baaeb42e7d175/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-13T23:52:38.147 DEBUG:teuthology.orchestra.run.smithi124:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-13T23:52:38.155 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-13T23:52:38.222 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2023-12-13T23:52:38.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2023-12-13T23:52:38.227 DEBUG:teuthology.orchestra.run.smithi124:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-13T23:52:38.293 INFO:teuthology.orchestra.run.smithi124.stdout:check_obsoletes = 1 2023-12-13T23:52:38.295 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum clean all 2023-12-13T23:52:38.460 INFO:teuthology.orchestra.run.smithi203.stdout:73 files removed 2023-12-13T23:52:38.487 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-12-13T23:52:38.540 INFO:teuthology.orchestra.run.smithi124.stdout:73 files removed 2023-12-13T23:52:38.567 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-12-13T23:52:39.269 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 172 kB/s | 78 kB 00:00 2023-12-13T23:52:39.306 INFO:teuthology.orchestra.run.smithi124.stdout:ceph packages for x86_64 186 kB/s | 78 kB 00:00 2023-12-13T23:52:39.603 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 33 kB/s | 9.5 kB 00:00 2023-12-13T23:52:39.615 INFO:teuthology.orchestra.run.smithi124.stdout:ceph noarch packages 37 kB/s | 9.5 kB 00:00 2023-12-13T23:52:39.932 INFO:teuthology.orchestra.run.smithi124.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2023-12-13T23:52:40.037 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2023-12-13T23:52:40.335 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-13T23:52:40.590 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-13T23:52:41.677 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2023-12-13T23:52:42.032 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2023-12-13T23:52:43.059 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 7.9 MB 00:01 2023-12-13T23:52:43.681 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 7.9 MB 00:01 2023-12-13T23:52:44.828 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - AppStream 26 MB/s | 18 MB 00:00 2023-12-13T23:52:45.698 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 18 MB/s | 18 MB 00:00 2023-12-13T23:52:49.782 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 5.8 MB 00:00 2023-12-13T23:52:50.137 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 5.8 MB 00:01 2023-12-13T23:52:51.067 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 15 kB 00:00 2023-12-13T23:52:51.921 INFO:teuthology.orchestra.run.smithi124.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 15 kB 00:00 2023-12-13T23:52:52.585 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 20 MB 00:01 2023-12-13T23:52:53.917 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2023-12-13T23:52:57.301 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-12-13T23:52:58.764 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2023-12-13T23:52:59.771 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-13T23:52:59.772 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-13T23:52:59.785 INFO:teuthology.orchestra.run.smithi203.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-12-13T23:52:59.786 INFO:teuthology.orchestra.run.smithi203.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-12-13T23:52:59.787 INFO:teuthology.orchestra.run.smithi203.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-12-13T23:52:59.788 INFO:teuthology.orchestra.run.smithi203.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-12-13T23:52:59.790 INFO:teuthology.orchestra.run.smithi203.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-12-13T23:52:59.793 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-12-13T23:52:59.794 INFO:teuthology.orchestra.run.smithi203.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-12-13T23:52:59.867 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2023-12-13T23:52:59.877 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2023-12-13T23:52:59.877 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2023-12-13T23:52:59.877 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-157.g021ac167.el9 ceph 6.5 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.2 M 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 859 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-157.g021ac167.el9 ceph 141 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.5 M 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 133 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 1.8 M 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 7.4 M 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 50 k 2023-12-13T23:52:59.878 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 8.6 M 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-157.g021ac167.el9 ceph 41 M 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 262 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 803 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 217 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell x86_64 2:19.0.0-157.g021ac167.el9 ceph 27 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 18 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 31 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 719 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 126 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-12-13T23:52:59.879 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-157.g021ac167.el9 ceph 161 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-157.g021ac167.el9 ceph 321 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 298 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 99 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 85 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 170 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2023-12-13T23:52:59.880 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.3 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-12-13T23:52:59.881 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 28 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-157.g021ac167.el9 ceph 2.1 M 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 243 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.4 M 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-157.g021ac167.el9 ceph 18 M 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 15 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-157.g021ac167.el9 ceph 24 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-13T23:52:59.882 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-157.g021ac167.el9 ceph 163 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 476 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.8 M 2023-12-13T23:52:59.883 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-13T23:52:59.884 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse x86_64 2:19.0.0-157.g021ac167.el9 ceph 45 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 129 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-13T23:52:59.885 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-13T23:52:59.886 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-13T23:52:59.887 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-13T23:52:59.888 INFO:teuthology.orchestra.run.smithi203.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout:=========================================================================================== 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout:Install 140 Packages 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 15 Packages 2023-12-13T23:52:59.889 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:52:59.890 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 205 M 2023-12-13T23:52:59.890 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2023-12-13T23:53:00.269 INFO:teuthology.orchestra.run.smithi203.stdout:(1/155): ceph-19.0.0-157.g021ac167.el9.x86_64.r 33 kB/s | 6.5 kB 00:00 2023-12-13T23:53:00.919 INFO:teuthology.orchestra.run.smithi203.stdout:(2/155): ceph-fuse-19.0.0-157.g021ac167.el9.x86 1.3 MB/s | 859 kB 00:00 2023-12-13T23:53:01.176 INFO:teuthology.orchestra.run.smithi124.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-13T23:53:01.177 INFO:teuthology.orchestra.run.smithi124.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-12-13T23:53:01.191 INFO:teuthology.orchestra.run.smithi124.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-12-13T23:53:01.192 INFO:teuthology.orchestra.run.smithi124.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-12-13T23:53:01.193 INFO:teuthology.orchestra.run.smithi124.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-12-13T23:53:01.194 INFO:teuthology.orchestra.run.smithi124.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-12-13T23:53:01.196 INFO:teuthology.orchestra.run.smithi124.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-12-13T23:53:01.199 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-12-13T23:53:01.200 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-12-13T23:53:01.236 INFO:teuthology.orchestra.run.smithi203.stdout:(3/155): ceph-common-19.0.0-157.g021ac167.el9.x 16 MB/s | 19 MB 00:01 2023-12-13T23:53:01.261 INFO:teuthology.orchestra.run.smithi203.stdout:(4/155): ceph-immutable-object-cache-19.0.0-157 412 kB/s | 141 kB 00:00 2023-12-13T23:53:01.273 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-12-13T23:53:01.292 INFO:teuthology.orchestra.run.smithi124.stdout:=========================================================================================== 2023-12-13T23:53:01.292 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout:=========================================================================================== 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph x86_64 2:19.0.0-157.g021ac167.el9 ceph 6.5 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base x86_64 2:19.0.0-157.g021ac167.el9 ceph 5.2 M 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 859 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-157.g021ac167.el9 ceph 141 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr x86_64 2:19.0.0-157.g021ac167.el9 ceph 1.5 M 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 133 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 1.8 M 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 7.4 M 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 50 k 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 8.6 M 2023-12-13T23:53:01.293 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test x86_64 2:19.0.0-157.g021ac167.el9 ceph 41 M 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 262 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 803 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 217 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-shell x86_64 2:19.0.0-157.g021ac167.el9 ceph 27 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 18 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel x86_64 0.190-1.el9 CentOS-AppStream 21 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 31 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 719 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel x86_64 2:19.0.0-157.g021ac167.el9 ceph 126 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-12-13T23:53:01.294 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel x86_64 1:3.0.7-25.el9 appstream 4.1 M 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs x86_64 2:19.0.0-157.g021ac167.el9 ceph 161 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados x86_64 2:19.0.0-157.g021ac167.el9 ceph 321 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 298 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw x86_64 2:19.0.0-157.g021ac167.el9 ceph 99 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse x86_64 2:19.0.0-157.g021ac167.el9 ceph 85 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd x86_64 2:19.0.0-157.g021ac167.el9 ceph 170 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 CentOS-AppStream 85 k 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrading: 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager x86_64 1:1.45.7-1.el9 baseos 2.3 M 2023-12-13T23:53:01.295 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-libnm x86_64 1:1.45.7-1.el9 baseos 1.8 M 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-team x86_64 1:1.45.7-1.el9 baseos 43 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-tui x86_64 1:1.45.7-1.el9 baseos 246 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-debuginfod-client x86_64 0.190-1.el9 CentOS-BaseOS 36 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf x86_64 0.190-1.el9 CentOS-BaseOS 192 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libs x86_64 0.190-1.el9 CentOS-BaseOS 255 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: librados2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.3 M 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 3.0 M 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: openssl x86_64 1:3.0.7-25.el9 baseos 1.2 M 2023-12-13T23:53:01.296 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-libs x86_64 1:3.0.7-25.el9 baseos 2.2 M 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 19 M 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 28 k 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds x86_64 2:19.0.0-157.g021ac167.el9 ceph 2.1 M 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 243 k 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.4 M 2023-12-13T23:53:01.297 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd x86_64 2:19.0.0-157.g021ac167.el9 ceph 18 M 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts noarch 2:19.0.0-157.g021ac167.el9 ceph-noarch 15 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux x86_64 2:19.0.0-157.g021ac167.el9 ceph 24 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite x86_64 2:19.0.0-157.g021ac167.el9 ceph 163 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-13T23:53:01.298 INFO:teuthology.orchestra.run.smithi124.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1 x86_64 2:19.0.0-157.g021ac167.el9 ceph 476 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2 x86_64 2:19.0.0-157.g021ac167.el9 ceph 4.8 M 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-13T23:53:01.299 INFO:teuthology.orchestra.run.smithi124.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse x86_64 2:19.0.0-157.g021ac167.el9 ceph 45 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common x86_64 2:19.0.0-157.g021ac167.el9 ceph 129 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2023-12-13T23:53:01.300 INFO:teuthology.orchestra.run.smithi124.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-13T23:53:01.301 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-13T23:53:01.302 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-13T23:53:01.303 INFO:teuthology.orchestra.run.smithi124.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout:Installing weak dependencies: 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:01.304 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout:=========================================================================================== 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout:Install 140 Packages 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrade 15 Packages 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 205 M 2023-12-13T23:53:01.305 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-12-13T23:53:01.379 INFO:teuthology.orchestra.run.smithi203.stdout:(5/155): ceph-mds-19.0.0-157.g021ac167.el9.x86_ 15 MB/s | 2.1 MB 00:00 2023-12-13T23:53:01.629 INFO:teuthology.orchestra.run.smithi203.stdout:(6/155): ceph-mon-19.0.0-157.g021ac167.el9.x86_ 18 MB/s | 4.4 MB 00:00 2023-12-13T23:53:01.974 INFO:teuthology.orchestra.run.smithi124.stdout:(1/155): ceph-19.0.0-157.g021ac167.el9.x86_64.r 19 kB/s | 6.5 kB 00:00 2023-12-13T23:53:02.287 INFO:teuthology.orchestra.run.smithi203.stdout:(7/155): ceph-mgr-19.0.0-157.g021ac167.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2023-12-13T23:53:02.490 INFO:teuthology.orchestra.run.smithi124.stdout:(2/155): ceph-fuse-19.0.0-157.g021ac167.el9.x86 1.6 MB/s | 859 kB 00:00 2023-12-13T23:53:02.532 INFO:teuthology.orchestra.run.smithi124.stdout:(3/155): ceph-immutable-object-cache-19.0.0-157 3.3 MB/s | 141 kB 00:00 2023-12-13T23:53:02.596 INFO:teuthology.orchestra.run.smithi203.stdout:(8/155): ceph-osd-19.0.0-157.g021ac167.el9.x86_ 18 MB/s | 18 MB 00:00 2023-12-13T23:53:02.671 INFO:teuthology.orchestra.run.smithi203.stdout:(9/155): ceph-base-19.0.0-157.g021ac167.el9.x86 2.0 MB/s | 5.2 MB 00:02 2023-12-13T23:53:02.688 INFO:teuthology.orchestra.run.smithi203.stdout:(10/155): ceph-selinux-19.0.0-157.g021ac167.el9 263 kB/s | 24 kB 00:00 2023-12-13T23:53:02.755 INFO:teuthology.orchestra.run.smithi203.stdout:(11/155): cephfs-mirror-19.0.0-157.g021ac167.el 3.2 MB/s | 217 kB 00:00 2023-12-13T23:53:02.813 INFO:teuthology.orchestra.run.smithi203.stdout:(12/155): cephfs-shell-19.0.0-157.g021ac167.el9 465 kB/s | 27 kB 00:00 2023-12-13T23:53:02.880 INFO:teuthology.orchestra.run.smithi203.stdout:(13/155): libcephfs-devel-19.0.0-157.g021ac167. 467 kB/s | 31 kB 00:00 2023-12-13T23:53:02.976 INFO:teuthology.orchestra.run.smithi124.stdout:(4/155): ceph-mds-19.0.0-157.g021ac167.el9.x86_ 4.7 MB/s | 2.1 MB 00:00 2023-12-13T23:53:02.981 INFO:teuthology.orchestra.run.smithi203.stdout:(14/155): libcephfs2-19.0.0-157.g021ac167.el9.x 7.0 MB/s | 719 kB 00:00 2023-12-13T23:53:03.064 INFO:teuthology.orchestra.run.smithi203.stdout:(15/155): libcephsqlite-19.0.0-157.g021ac167.el 1.9 MB/s | 163 kB 00:00 2023-12-13T23:53:03.148 INFO:teuthology.orchestra.run.smithi203.stdout:(16/155): librados-devel-19.0.0-157.g021ac167.e 1.5 MB/s | 126 kB 00:00 2023-12-13T23:53:03.248 INFO:teuthology.orchestra.run.smithi203.stdout:(17/155): libradosstriper1-19.0.0-157.g021ac167 4.7 MB/s | 476 kB 00:00 2023-12-13T23:53:03.385 INFO:teuthology.orchestra.run.smithi124.stdout:(5/155): ceph-mgr-19.0.0-157.g021ac167.el9.x86_ 3.6 MB/s | 1.5 MB 00:00 2023-12-13T23:53:03.484 INFO:teuthology.orchestra.run.smithi124.stdout:(6/155): ceph-base-19.0.0-157.g021ac167.el9.x86 2.8 MB/s | 5.2 MB 00:01 2023-12-13T23:53:03.999 INFO:teuthology.orchestra.run.smithi203.stdout:(18/155): librgw2-19.0.0-157.g021ac167.el9.x86_ 6.4 MB/s | 4.8 MB 00:00 2023-12-13T23:53:04.032 INFO:teuthology.orchestra.run.smithi203.stdout:(19/155): python3-ceph-argparse-19.0.0-157.g021 1.3 MB/s | 45 kB 00:00 2023-12-13T23:53:04.083 INFO:teuthology.orchestra.run.smithi203.stdout:(20/155): python3-ceph-common-19.0.0-157.g021ac 2.5 MB/s | 129 kB 00:00 2023-12-13T23:53:04.125 INFO:teuthology.orchestra.run.smithi203.stdout:(21/155): python3-cephfs-19.0.0-157.g021ac167.e 3.8 MB/s | 161 kB 00:00 2023-12-13T23:53:04.159 INFO:teuthology.orchestra.run.smithi124.stdout:(7/155): ceph-common-19.0.0-157.g021ac167.el9.x 7.5 MB/s | 19 MB 00:02 2023-12-13T23:53:04.183 INFO:teuthology.orchestra.run.smithi203.stdout:(22/155): python3-rados-19.0.0-157.g021ac167.el 5.4 MB/s | 321 kB 00:00 2023-12-13T23:53:04.259 INFO:teuthology.orchestra.run.smithi203.stdout:(23/155): python3-rbd-19.0.0-157.g021ac167.el9. 3.9 MB/s | 298 kB 00:00 2023-12-13T23:53:04.334 INFO:teuthology.orchestra.run.smithi203.stdout:(24/155): python3-rgw-19.0.0-157.g021ac167.el9. 1.3 MB/s | 99 kB 00:00 2023-12-13T23:53:04.359 INFO:teuthology.orchestra.run.smithi124.stdout:(8/155): ceph-mon-19.0.0-157.g021ac167.el9.x86_ 4.5 MB/s | 4.4 MB 00:00 2023-12-13T23:53:04.392 INFO:teuthology.orchestra.run.smithi203.stdout:(25/155): rbd-fuse-19.0.0-157.g021ac167.el9.x86 1.4 MB/s | 85 kB 00:00 2023-12-13T23:53:04.418 INFO:teuthology.orchestra.run.smithi124.stdout:(9/155): ceph-selinux-19.0.0-157.g021ac167.el9. 415 kB/s | 24 kB 00:00 2023-12-13T23:53:04.826 INFO:teuthology.orchestra.run.smithi203.stdout:(26/155): rbd-mirror-19.0.0-157.g021ac167.el9.x 6.9 MB/s | 3.0 MB 00:00 2023-12-13T23:53:04.910 INFO:teuthology.orchestra.run.smithi203.stdout:(27/155): rbd-nbd-19.0.0-157.g021ac167.el9.x86_ 2.0 MB/s | 170 kB 00:00 2023-12-13T23:53:04.960 INFO:teuthology.orchestra.run.smithi203.stdout:(28/155): ceph-grafana-dashboards-19.0.0-157.g0 565 kB/s | 28 kB 00:00 2023-12-13T23:53:05.001 INFO:teuthology.orchestra.run.smithi124.stdout:(10/155): ceph-radosgw-19.0.0-157.g021ac167.el9 10 MB/s | 8.6 MB 00:00 2023-12-13T23:53:05.010 INFO:teuthology.orchestra.run.smithi203.stdout:(29/155): ceph-mgr-cephadm-19.0.0-157.g021ac167 2.6 MB/s | 133 kB 00:00 2023-12-13T23:53:05.093 INFO:teuthology.orchestra.run.smithi124.stdout:(11/155): cephfs-mirror-19.0.0-157.g021ac167.el 2.3 MB/s | 217 kB 00:00 2023-12-13T23:53:05.193 INFO:teuthology.orchestra.run.smithi124.stdout:(12/155): cephfs-shell-19.0.0-157.g021ac167.el9 270 kB/s | 27 kB 00:00 2023-12-13T23:53:05.194 INFO:teuthology.orchestra.run.smithi203.stdout:(30/155): ceph-mgr-dashboard-19.0.0-157.g021ac1 9.6 MB/s | 1.8 MB 00:00 2023-12-13T23:53:05.269 INFO:teuthology.orchestra.run.smithi124.stdout:(13/155): libcephfs-devel-19.0.0-157.g021ac167. 415 kB/s | 31 kB 00:00 2023-12-13T23:53:05.377 INFO:teuthology.orchestra.run.smithi124.stdout:(14/155): libcephfs2-19.0.0-157.g021ac167.el9.x 6.5 MB/s | 719 kB 00:00 2023-12-13T23:53:05.471 INFO:teuthology.orchestra.run.smithi124.stdout:(15/155): libcephsqlite-19.0.0-157.g021ac167.el 1.7 MB/s | 163 kB 00:00 2023-12-13T23:53:05.527 INFO:teuthology.orchestra.run.smithi203.stdout:(31/155): ceph-radosgw-19.0.0-157.g021ac167.el9 2.7 MB/s | 8.6 MB 00:03 2023-12-13T23:53:05.580 INFO:teuthology.orchestra.run.smithi124.stdout:(16/155): librados-devel-19.0.0-157.g021ac167.e 1.1 MB/s | 126 kB 00:00 2023-12-13T23:53:05.694 INFO:teuthology.orchestra.run.smithi203.stdout:(32/155): ceph-mgr-modules-core-19.0.0-157.g021 1.4 MB/s | 243 kB 00:00 2023-12-13T23:53:05.696 INFO:teuthology.orchestra.run.smithi124.stdout:(17/155): libradosstriper1-19.0.0-157.g021ac167 4.0 MB/s | 476 kB 00:00 2023-12-13T23:53:05.757 INFO:teuthology.orchestra.run.smithi203.stdout:(33/155): ceph-mgr-rook-19.0.0-157.g021ac167.el 799 kB/s | 50 kB 00:00 2023-12-13T23:53:05.857 INFO:teuthology.orchestra.run.smithi203.stdout:(34/155): ceph-prometheus-alerts-19.0.0-157.g02 151 kB/s | 15 kB 00:00 2023-12-13T23:53:06.016 INFO:teuthology.orchestra.run.smithi203.stdout:(35/155): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2023-12-13T23:53:06.058 INFO:teuthology.orchestra.run.smithi203.stdout:(36/155): ceph-volume-19.0.0-157.g021ac167.el9. 1.3 MB/s | 262 kB 00:00 2023-12-13T23:53:06.142 INFO:teuthology.orchestra.run.smithi203.stdout:(37/155): cephfs-top-19.0.0-157.g021ac167.el9.n 218 kB/s | 18 kB 00:00 2023-12-13T23:53:06.184 INFO:teuthology.orchestra.run.smithi203.stdout:(38/155): cephadm-19.0.0-157.g021ac167.el9.noar 4.7 MB/s | 803 kB 00:00 2023-12-13T23:53:06.205 INFO:teuthology.orchestra.run.smithi124.stdout:(18/155): librgw2-19.0.0-157.g021ac167.el9.x86_ 9.4 MB/s | 4.8 MB 00:00 2023-12-13T23:53:06.273 INFO:teuthology.orchestra.run.smithi124.stdout:(19/155): python3-ceph-argparse-19.0.0-157.g021 677 kB/s | 45 kB 00:00 2023-12-13T23:53:06.326 INFO:teuthology.orchestra.run.smithi203.stdout:(39/155): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 662 kB 00:00 2023-12-13T23:53:06.340 INFO:teuthology.orchestra.run.smithi124.stdout:(20/155): python3-ceph-common-19.0.0-157.g021ac 1.9 MB/s | 129 kB 00:00 2023-12-13T23:53:06.368 INFO:teuthology.orchestra.run.smithi203.stdout:(40/155): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 920 kB 00:00 2023-12-13T23:53:06.393 INFO:teuthology.orchestra.run.smithi203.stdout:(41/155): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2023-12-13T23:53:06.410 INFO:teuthology.orchestra.run.smithi203.stdout:(42/155): elfutils-libelf-devel-0.190-1.el9.x86 519 kB/s | 21 kB 00:00 2023-12-13T23:53:06.415 INFO:teuthology.orchestra.run.smithi124.stdout:(21/155): python3-cephfs-19.0.0-157.g021ac167.e 2.1 MB/s | 161 kB 00:00 2023-12-13T23:53:06.437 INFO:teuthology.orchestra.run.smithi203.stdout:(43/155): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2023-12-13T23:53:06.461 INFO:teuthology.orchestra.run.smithi203.stdout:(44/155): flexiblas-3.0.4-8.el9.x86_64.rpm 635 kB/s | 32 kB 00:00 2023-12-13T23:53:06.499 INFO:teuthology.orchestra.run.smithi124.stdout:(22/155): python3-rados-19.0.0-157.g021ac167.el 3.8 MB/s | 321 kB 00:00 2023-12-13T23:53:06.536 INFO:teuthology.orchestra.run.smithi203.stdout:(45/155): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2023-12-13T23:53:06.553 INFO:teuthology.orchestra.run.smithi203.stdout:(46/155): flexiblas-openblas-openmp-3.0.4-8.el9 191 kB/s | 17 kB 00:00 2023-12-13T23:53:06.582 INFO:teuthology.orchestra.run.smithi124.stdout:(23/155): python3-rbd-19.0.0-157.g021ac167.el9. 3.5 MB/s | 298 kB 00:00 2023-12-13T23:53:06.587 INFO:teuthology.orchestra.run.smithi203.stdout:(47/155): libaio-devel-0.3.111-13.el9.x86_64.rp 303 kB/s | 10 kB 00:00 2023-12-13T23:53:06.612 INFO:teuthology.orchestra.run.smithi203.stdout:(48/155): jq-1.6-16.el9.x86_64.rpm 3.1 MB/s | 186 kB 00:00 2023-12-13T23:53:06.637 INFO:teuthology.orchestra.run.smithi203.stdout:(49/155): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-13T23:53:06.657 INFO:teuthology.orchestra.run.smithi124.stdout:(24/155): python3-rgw-19.0.0-157.g021ac167.el9. 1.3 MB/s | 99 kB 00:00 2023-12-13T23:53:06.663 INFO:teuthology.orchestra.run.smithi203.stdout:(50/155): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2023-12-13T23:53:06.688 INFO:teuthology.orchestra.run.smithi203.stdout:(51/155): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-13T23:53:06.713 INFO:teuthology.orchestra.run.smithi203.stdout:(52/155): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2023-12-13T23:53:06.733 INFO:teuthology.orchestra.run.smithi124.stdout:(25/155): rbd-fuse-19.0.0-157.g021ac167.el9.x86 1.1 MB/s | 85 kB 00:00 2023-12-13T23:53:06.738 INFO:teuthology.orchestra.run.smithi203.stdout:(53/155): libuuid-devel-2.37.4-15.el9.x86_64.rp 497 kB/s | 24 kB 00:00 2023-12-13T23:53:06.764 INFO:teuthology.orchestra.run.smithi203.stdout:(54/155): libzstd-devel-1.5.1-2.el9.x86_64.rpm 923 kB/s | 46 kB 00:00 2023-12-13T23:53:06.789 INFO:teuthology.orchestra.run.smithi203.stdout:(55/155): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-12-13T23:53:06.814 INFO:teuthology.orchestra.run.smithi203.stdout:(56/155): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-13T23:53:06.840 INFO:teuthology.orchestra.run.smithi203.stdout:(57/155): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-13T23:53:06.866 INFO:teuthology.orchestra.run.smithi203.stdout:(58/155): openblas-0.3.21-2.el9.x86_64.rpm 684 kB/s | 34 kB 00:00 2023-12-13T23:53:06.940 INFO:teuthology.orchestra.run.smithi203.stdout:(59/155): openblas-openmp-0.3.21-2.el9.x86_64.r 47 MB/s | 4.7 MB 00:00 2023-12-13T23:53:06.982 INFO:teuthology.orchestra.run.smithi203.stdout:(60/155): postgresql-13.11-1.el9.x86_64.rpm 13 MB/s | 1.5 MB 00:00 2023-12-13T23:53:07.016 INFO:teuthology.orchestra.run.smithi203.stdout:(61/155): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2023-12-13T23:53:07.016 INFO:teuthology.orchestra.run.smithi124.stdout:(26/155): rbd-mirror-19.0.0-157.g021ac167.el9.x 11 MB/s | 3.0 MB 00:00 2023-12-13T23:53:07.041 INFO:teuthology.orchestra.run.smithi203.stdout:(62/155): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2023-12-13T23:53:07.092 INFO:teuthology.orchestra.run.smithi124.stdout:(27/155): rbd-nbd-19.0.0-157.g021ac167.el9.x86_ 2.2 MB/s | 170 kB 00:00 2023-12-13T23:53:07.124 INFO:teuthology.orchestra.run.smithi203.stdout:(63/155): postgresql-server-13.11-1.el9.x86_64. 53 MB/s | 5.7 MB 00:00 2023-12-13T23:53:07.141 INFO:teuthology.orchestra.run.smithi203.stdout:(64/155): python3-attrs-20.3.0-7.el9.noarch.rpm 799 kB/s | 78 kB 00:00 2023-12-13T23:53:07.159 INFO:teuthology.orchestra.run.smithi124.stdout:(28/155): ceph-grafana-dashboards-19.0.0-157.g0 423 kB/s | 28 kB 00:00 2023-12-13T23:53:07.175 INFO:teuthology.orchestra.run.smithi203.stdout:(65/155): python3-jinja2-2.11.3-4.el9.noarch.rp 6.8 MB/s | 228 kB 00:00 2023-12-13T23:53:07.226 INFO:teuthology.orchestra.run.smithi124.stdout:(29/155): ceph-mgr-cephadm-19.0.0-157.g021ac167 1.9 MB/s | 133 kB 00:00 2023-12-13T23:53:07.259 INFO:teuthology.orchestra.run.smithi203.stdout:(66/155): python3-babel-2.9.1-2.el9.noarch.rpm 49 MB/s | 5.8 MB 00:00 2023-12-13T23:53:07.284 INFO:teuthology.orchestra.run.smithi203.stdout:(67/155): python3-jsonpatch-1.21-16.el9.noarch. 231 kB/s | 24 kB 00:00 2023-12-13T23:53:07.317 INFO:teuthology.orchestra.run.smithi203.stdout:(68/155): python3-jsonpointer-2.0-4.el9.noarch. 547 kB/s | 18 kB 00:00 2023-12-13T23:53:07.343 INFO:teuthology.orchestra.run.smithi203.stdout:(69/155): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2023-12-13T23:53:07.368 INFO:teuthology.orchestra.run.smithi203.stdout:(70/155): python3-markupsafe-1.1.1-12.el9.x86_6 632 kB/s | 32 kB 00:00 2023-12-13T23:53:07.393 INFO:teuthology.orchestra.run.smithi124.stdout:(30/155): ceph-mgr-dashboard-19.0.0-157.g021ac1 11 MB/s | 1.8 MB 00:00 2023-12-13T23:53:07.452 INFO:teuthology.orchestra.run.smithi203.stdout:(71/155): python3-numpy-1.20.1-5.el9.x86_64.rpm 44 MB/s | 4.8 MB 00:00 2023-12-13T23:53:07.477 INFO:teuthology.orchestra.run.smithi203.stdout:(72/155): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 217 kB 00:00 2023-12-13T23:53:07.502 INFO:teuthology.orchestra.run.smithi203.stdout:(73/155): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2023-12-13T23:53:07.528 INFO:teuthology.orchestra.run.smithi203.stdout:(74/155): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2023-12-13T23:53:07.551 INFO:teuthology.orchestra.run.smithi124.stdout:(31/155): ceph-osd-19.0.0-157.g021ac167.el9.x86 4.4 MB/s | 18 MB 00:04 2023-12-13T23:53:07.553 INFO:teuthology.orchestra.run.smithi203.stdout:(75/155): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2023-12-13T23:53:07.579 INFO:teuthology.orchestra.run.smithi203.stdout:(76/155): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-13T23:53:07.603 INFO:teuthology.orchestra.run.smithi203.stdout:(77/155): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2023-12-13T23:53:07.629 INFO:teuthology.orchestra.run.smithi203.stdout:(78/155): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2023-12-13T23:53:07.686 INFO:teuthology.orchestra.run.smithi124.stdout:(32/155): ceph-mgr-modules-core-19.0.0-157.g021 1.8 MB/s | 243 kB 00:00 2023-12-13T23:53:07.752 INFO:teuthology.orchestra.run.smithi124.stdout:(33/155): ceph-mgr-rook-19.0.0-157.g021ac167.el 753 kB/s | 50 kB 00:00 2023-12-13T23:53:07.819 INFO:teuthology.orchestra.run.smithi124.stdout:(34/155): ceph-prometheus-alerts-19.0.0-157.g02 226 kB/s | 15 kB 00:00 2023-12-13T23:53:07.854 INFO:teuthology.orchestra.run.smithi203.stdout:(79/155): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2023-12-13T23:53:07.871 INFO:teuthology.orchestra.run.smithi203.stdout:(80/155): python3-toml-0.10.2-6.el9.noarch.rpm 161 kB/s | 39 kB 00:00 2023-12-13T23:53:07.888 INFO:teuthology.orchestra.run.smithi203.stdout:(81/155): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2023-12-13T23:53:07.913 INFO:teuthology.orchestra.run.smithi203.stdout:(82/155): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2023-12-13T23:53:07.930 INFO:teuthology.orchestra.run.smithi203.stdout:(83/155): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2023-12-13T23:53:07.947 INFO:teuthology.orchestra.run.smithi203.stdout:(84/155): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 2.5 MB/s | 85 kB 00:00 2023-12-13T23:53:07.953 INFO:teuthology.orchestra.run.smithi124.stdout:(35/155): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2023-12-13T23:53:07.971 INFO:teuthology.orchestra.run.smithi203.stdout:(85/155): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2023-12-13T23:53:07.978 INFO:teuthology.orchestra.run.smithi124.stdout:(36/155): ceph-volume-19.0.0-157.g021ac167.el9. 1.6 MB/s | 262 kB 00:00 2023-12-13T23:53:07.996 INFO:teuthology.orchestra.run.smithi203.stdout:(86/155): zlib-devel-1.2.11-40.el9.x86_64.rpm 907 kB/s | 44 kB 00:00 2023-12-13T23:53:08.021 INFO:teuthology.orchestra.run.smithi203.stdout:(87/155): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2023-12-13T23:53:08.045 INFO:teuthology.orchestra.run.smithi124.stdout:(37/155): cephadm-19.0.0-157.g021ac167.el9.noar 8.5 MB/s | 803 kB 00:00 2023-12-13T23:53:08.055 INFO:teuthology.orchestra.run.smithi203.stdout:(88/155): libgfortran-11.4.1-2.3.el9.x86_64.rpm 13 MB/s | 803 kB 00:00 2023-12-13T23:53:08.070 INFO:teuthology.orchestra.run.smithi124.stdout:(38/155): cephfs-top-19.0.0-157.g021ac167.el9.n 198 kB/s | 18 kB 00:00 2023-12-13T23:53:08.080 INFO:teuthology.orchestra.run.smithi203.stdout:(89/155): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2023-12-13T23:53:08.106 INFO:teuthology.orchestra.run.smithi203.stdout:(90/155): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-13T23:53:08.140 INFO:teuthology.orchestra.run.smithi203.stdout:(91/155): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2023-12-13T23:53:08.165 INFO:teuthology.orchestra.run.smithi203.stdout:(92/155): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2023-12-13T23:53:08.190 INFO:teuthology.orchestra.run.smithi203.stdout:(93/155): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2023-12-13T23:53:08.212 INFO:teuthology.orchestra.run.smithi124.stdout:(39/155): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2023-12-13T23:53:08.216 INFO:teuthology.orchestra.run.smithi203.stdout:(94/155): python3-requests-2.25.1-7.el9.noarch. 2.2 MB/s | 113 kB 00:00 2023-12-13T23:53:08.237 INFO:teuthology.orchestra.run.smithi124.stdout:(40/155): bison-3.7.4-5.el9.x86_64.rpm 5.4 MB/s | 920 kB 00:00 2023-12-13T23:53:08.241 INFO:teuthology.orchestra.run.smithi203.stdout:(95/155): python3-urllib3-1.26.5-4.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2023-12-13T23:53:08.263 INFO:teuthology.orchestra.run.smithi124.stdout:(41/155): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2023-12-13T23:53:08.280 INFO:teuthology.orchestra.run.smithi124.stdout:(42/155): elfutils-libelf-devel-0.190-1.el9.x86 515 kB/s | 21 kB 00:00 2023-12-13T23:53:08.305 INFO:teuthology.orchestra.run.smithi124.stdout:(43/155): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2023-12-13T23:53:08.322 INFO:teuthology.orchestra.run.smithi124.stdout:(44/155): flexiblas-3.0.4-8.el9.x86_64.rpm 763 kB/s | 32 kB 00:00 2023-12-13T23:53:08.406 INFO:teuthology.orchestra.run.smithi124.stdout:(45/155): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2023-12-13T23:53:08.422 INFO:teuthology.orchestra.run.smithi124.stdout:(46/155): flexiblas-openblas-openmp-3.0.4-8.el9 173 kB/s | 17 kB 00:00 2023-12-13T23:53:08.448 INFO:teuthology.orchestra.run.smithi124.stdout:(47/155): libaio-devel-0.3.111-13.el9.x86_64.rp 404 kB/s | 10 kB 00:00 2023-12-13T23:53:08.473 INFO:teuthology.orchestra.run.smithi124.stdout:(48/155): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2023-12-13T23:53:08.491 INFO:teuthology.orchestra.run.smithi203.stdout:(96/155): python3-devel-3.9.17-2.el9.x86_64.rpm 982 kB/s | 245 kB 00:00 2023-12-13T23:53:08.498 INFO:teuthology.orchestra.run.smithi124.stdout:(49/155): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2023-12-13T23:53:08.524 INFO:teuthology.orchestra.run.smithi124.stdout:(50/155): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2023-12-13T23:53:08.549 INFO:teuthology.orchestra.run.smithi124.stdout:(51/155): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2023-12-13T23:53:08.583 INFO:teuthology.orchestra.run.smithi124.stdout:(52/155): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2023-12-13T23:53:08.608 INFO:teuthology.orchestra.run.smithi124.stdout:(53/155): libuuid-devel-2.37.4-15.el9.x86_64.rp 423 kB/s | 24 kB 00:00 2023-12-13T23:53:08.633 INFO:teuthology.orchestra.run.smithi124.stdout:(54/155): libzstd-devel-1.5.1-2.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2023-12-13T23:53:08.658 INFO:teuthology.orchestra.run.smithi124.stdout:(55/155): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.5 MB/s | 282 kB 00:00 2023-12-13T23:53:08.684 INFO:teuthology.orchestra.run.smithi124.stdout:(56/155): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-12-13T23:53:08.708 INFO:teuthology.orchestra.run.smithi203.stdout:(97/155): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.4 MB/s | 309 kB 00:00 2023-12-13T23:53:08.718 INFO:teuthology.orchestra.run.smithi124.stdout:(57/155): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2023-12-13T23:53:08.743 INFO:teuthology.orchestra.run.smithi124.stdout:(58/155): openblas-0.3.21-2.el9.x86_64.rpm 584 kB/s | 34 kB 00:00 2023-12-13T23:53:08.810 INFO:teuthology.orchestra.run.smithi124.stdout:(59/155): openblas-openmp-0.3.21-2.el9.x86_64.r 52 MB/s | 4.7 MB 00:00 2023-12-13T23:53:08.852 INFO:teuthology.orchestra.run.smithi124.stdout:(60/155): postgresql-13.11-1.el9.x86_64.rpm 14 MB/s | 1.5 MB 00:00 2023-12-13T23:53:08.885 INFO:teuthology.orchestra.run.smithi124.stdout:(61/155): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2023-12-13T23:53:08.910 INFO:teuthology.orchestra.run.smithi124.stdout:(62/155): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2023-12-13T23:53:09.010 INFO:teuthology.orchestra.run.smithi124.stdout:(63/155): postgresql-server-13.11-1.el9.x86_64. 57 MB/s | 5.7 MB 00:00 2023-12-13T23:53:09.011 INFO:teuthology.orchestra.run.smithi203.stdout:(98/155): libarrow-9.0.0-10.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2023-12-13T23:53:09.011 INFO:teuthology.orchestra.run.smithi203.stdout:(99/155): libarrow-doc-9.0.0-10.el9.noarch.rpm 448 kB/s | 26 kB 00:00 2023-12-13T23:53:09.011 INFO:teuthology.orchestra.run.smithi124.stdout:(64/155): python3-attrs-20.3.0-7.el9.noarch.rpm 792 kB/s | 78 kB 00:00 2023-12-13T23:53:09.051 INFO:teuthology.orchestra.run.smithi203.stdout:(100/155): liboath-2.6.7-2.el9.x86_64.rpm 971 kB/s | 49 kB 00:00 2023-12-13T23:53:09.053 INFO:teuthology.orchestra.run.smithi124.stdout:(65/155): python3-jinja2-2.11.3-4.el9.noarch.rp 5.4 MB/s | 228 kB 00:00 2023-12-13T23:53:09.101 INFO:teuthology.orchestra.run.smithi203.stdout:(101/155): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-12-13T23:53:09.129 INFO:teuthology.orchestra.run.smithi124.stdout:(66/155): python3-babel-2.9.1-2.el9.noarch.rpm 49 MB/s | 5.8 MB 00:00 2023-12-13T23:53:09.172 INFO:teuthology.orchestra.run.smithi124.stdout:(67/155): python3-jsonpatch-1.21-16.el9.noarch. 209 kB/s | 24 kB 00:00 2023-12-13T23:53:09.193 INFO:teuthology.orchestra.run.smithi203.stdout:(102/155): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2023-12-13T23:53:09.197 INFO:teuthology.orchestra.run.smithi124.stdout:(68/155): python3-jsonpointer-2.0-4.el9.noarch. 263 kB/s | 18 kB 00:00 2023-12-13T23:53:09.215 INFO:teuthology.orchestra.run.smithi124.stdout:(69/155): python3-mako-1.1.4-6.el9.noarch.rpm 3.3 MB/s | 144 kB 00:00 2023-12-13T23:53:09.232 INFO:teuthology.orchestra.run.smithi124.stdout:(70/155): python3-markupsafe-1.1.1-12.el9.x86_6 941 kB/s | 32 kB 00:00 2023-12-13T23:53:09.252 INFO:teuthology.orchestra.run.smithi203.stdout:(103/155): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2023-12-13T23:53:09.332 INFO:teuthology.orchestra.run.smithi124.stdout:(71/155): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 4.8 MB 00:00 2023-12-13T23:53:09.352 INFO:teuthology.orchestra.run.smithi203.stdout:(104/155): python3-asyncssh-2.13.2-3.el9.noarch 5.4 MB/s | 548 kB 00:00 2023-12-13T23:53:09.357 INFO:teuthology.orchestra.run.smithi124.stdout:(72/155): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2023-12-13T23:53:09.382 INFO:teuthology.orchestra.run.smithi124.stdout:(73/155): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2023-12-13T23:53:09.402 INFO:teuthology.orchestra.run.smithi203.stdout:(105/155): python3-bcrypt-3.2.2-1.el9.x86_64.rp 868 kB/s | 43 kB 00:00 2023-12-13T23:53:09.416 INFO:teuthology.orchestra.run.smithi124.stdout:(74/155): python3-prettytable-0.7.2-27.el9.noar 709 kB/s | 41 kB 00:00 2023-12-13T23:53:09.441 INFO:teuthology.orchestra.run.smithi124.stdout:(75/155): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.2 MB/s | 132 kB 00:00 2023-12-13T23:53:09.455 INFO:teuthology.orchestra.run.smithi203.stdout:(106/155): python3-cachetools-4.2.4-1.el9.noarc 643 kB/s | 32 kB 00:00 2023-12-13T23:53:09.467 INFO:teuthology.orchestra.run.smithi124.stdout:(76/155): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2023-12-13T23:53:09.492 INFO:teuthology.orchestra.run.smithi124.stdout:(77/155): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2023-12-13T23:53:09.495 INFO:teuthology.orchestra.run.smithi203.stdout:(107/155): python3-certifi-2021.10.8-2.el9.noar 361 kB/s | 15 kB 00:00 2023-12-13T23:53:09.517 INFO:teuthology.orchestra.run.smithi124.stdout:(78/155): python3-requests-oauthlib-1.3.0-12.el 857 kB/s | 43 kB 00:00 2023-12-13T23:53:09.554 INFO:teuthology.orchestra.run.smithi203.stdout:(108/155): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2023-12-13T23:53:09.612 INFO:teuthology.orchestra.run.smithi203.stdout:(109/155): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2023-12-13T23:53:09.663 INFO:teuthology.orchestra.run.smithi203.stdout:(110/155): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2023-12-13T23:53:09.713 INFO:teuthology.orchestra.run.smithi203.stdout:(111/155): python3-colorama-0.4.4-12.el9.noarch 707 kB/s | 35 kB 00:00 2023-12-13T23:53:09.772 INFO:teuthology.orchestra.run.smithi203.stdout:(112/155): openssl-devel-3.0.7-25.el9.x86_64.rp 2.7 MB/s | 4.1 MB 00:01 2023-12-13T23:53:09.813 INFO:teuthology.orchestra.run.smithi203.stdout:(113/155): python3-google-auth-2.23.4-1.el9.noa 1.9 MB/s | 197 kB 00:00 2023-12-13T23:53:09.839 INFO:teuthology.orchestra.run.smithi203.stdout:(114/155): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2023-12-13T23:53:09.856 INFO:teuthology.orchestra.run.smithi203.stdout:(115/155): python3-jaraco-classes-3.2.1-5.el9.n 422 kB/s | 18 kB 00:00 2023-12-13T23:53:09.890 INFO:teuthology.orchestra.run.smithi203.stdout:(116/155): python3-jaraco-collections-3.0.0-8.e 458 kB/s | 23 kB 00:00 2023-12-13T23:53:09.901 INFO:teuthology.orchestra.run.smithi124.stdout:(79/155): ceph-test-19.0.0-157.g021ac167.el9.x8 7.5 MB/s | 41 MB 00:05 2023-12-13T23:53:09.915 INFO:teuthology.orchestra.run.smithi203.stdout:(117/155): python3-jaraco-functools-3.5.0-2.el9 330 kB/s | 19 kB 00:00 2023-12-13T23:53:09.932 INFO:teuthology.orchestra.run.smithi203.stdout:(118/155): python3-jaraco-text-3.2.0-6.el9.noar 471 kB/s | 20 kB 00:00 2023-12-13T23:53:09.949 INFO:teuthology.orchestra.run.smithi203.stdout:(119/155): python3-jwt+crypto-2.4.0-1.el9.noarc 270 kB/s | 9.0 kB 00:00 2023-12-13T23:53:09.966 INFO:teuthology.orchestra.run.smithi203.stdout:(120/155): python3-jwt-2.4.0-1.el9.noarch.rpm 1.2 MB/s | 41 kB 00:00 2023-12-13T23:53:10.001 INFO:teuthology.orchestra.run.smithi124.stdout:(80/155): python3-toml-0.10.2-6.el9.noarch.rpm 80 kB/s | 39 kB 00:00 2023-12-13T23:53:10.058 INFO:teuthology.orchestra.run.smithi203.stdout:(121/155): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2023-12-13T23:53:10.075 INFO:teuthology.orchestra.run.smithi203.stdout:(122/155): python3-logutils-0.3.5-21.el9.noarch 426 kB/s | 46 kB 00:00 2023-12-13T23:53:10.108 INFO:teuthology.orchestra.run.smithi203.stdout:(123/155): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2023-12-13T23:53:10.134 INFO:teuthology.orchestra.run.smithi203.stdout:(124/155): python3-natsort-7.1.1-5.el9.noarch.r 985 kB/s | 58 kB 00:00 2023-12-13T23:53:10.151 INFO:teuthology.orchestra.run.smithi124.stdout:(81/155): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2023-12-13T23:53:10.192 INFO:teuthology.orchestra.run.smithi203.stdout:(125/155): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2023-12-13T23:53:10.193 INFO:teuthology.orchestra.run.smithi124.stdout:(82/155): python3-wcwidth-0.2.5-8.el9.noarch.rp 142 kB/s | 41 kB 00:00 2023-12-13T23:53:10.218 INFO:teuthology.orchestra.run.smithi203.stdout:(126/155): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2023-12-13T23:53:10.218 INFO:teuthology.orchestra.run.smithi124.stdout:(83/155): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2023-12-13T23:53:10.243 INFO:teuthology.orchestra.run.smithi203.stdout:(127/155): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2023-12-13T23:53:10.243 INFO:teuthology.orchestra.run.smithi124.stdout:(84/155): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2023-12-13T23:53:10.268 INFO:teuthology.orchestra.run.smithi124.stdout:(85/155): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 1.1 MB/s | 85 kB 00:00 2023-12-13T23:53:10.277 INFO:teuthology.orchestra.run.smithi203.stdout:(128/155): python3-pyperclip-1.8.2-2.el9.noarch 481 kB/s | 28 kB 00:00 2023-12-13T23:53:10.294 INFO:teuthology.orchestra.run.smithi124.stdout:(86/155): xmlstarlet-1.6.1-20.el9.x86_64.rpm 835 kB/s | 63 kB 00:00 2023-12-13T23:53:10.302 INFO:teuthology.orchestra.run.smithi203.stdout:(129/155): python3-repoze-lru-0.7-16.el9.noarch 524 kB/s | 31 kB 00:00 2023-12-13T23:53:10.319 INFO:teuthology.orchestra.run.smithi124.stdout:(87/155): zlib-devel-1.2.11-40.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2023-12-13T23:53:10.344 INFO:teuthology.orchestra.run.smithi124.stdout:(88/155): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2023-12-13T23:53:10.369 INFO:teuthology.orchestra.run.smithi203.stdout:(130/155): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2023-12-13T23:53:10.386 INFO:teuthology.orchestra.run.smithi124.stdout:(89/155): libgfortran-11.4.1-2.3.el9.x86_64.rpm 8.5 MB/s | 803 kB 00:00 2023-12-13T23:53:10.394 INFO:teuthology.orchestra.run.smithi203.stdout:(131/155): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2023-12-13T23:53:10.412 INFO:teuthology.orchestra.run.smithi124.stdout:(90/155): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2023-12-13T23:53:10.428 INFO:teuthology.orchestra.run.smithi203.stdout:(132/155): python3-tempora-5.0.0-2.el9.noarch.r 613 kB/s | 36 kB 00:00 2023-12-13T23:53:10.436 INFO:teuthology.orchestra.run.smithi124.stdout:(91/155): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.6 MB/s | 241 kB 00:00 2023-12-13T23:53:10.453 INFO:teuthology.orchestra.run.smithi203.stdout:(133/155): python3-typing-extensions-4.4.0-2.el 869 kB/s | 51 kB 00:00 2023-12-13T23:53:10.470 INFO:teuthology.orchestra.run.smithi124.stdout:(92/155): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2023-12-13T23:53:10.496 INFO:teuthology.orchestra.run.smithi124.stdout:(93/155): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2023-12-13T23:53:10.521 INFO:teuthology.orchestra.run.smithi124.stdout:(94/155): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2023-12-13T23:53:10.546 INFO:teuthology.orchestra.run.smithi124.stdout:(95/155): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-12-13T23:53:10.571 INFO:teuthology.orchestra.run.smithi124.stdout:(96/155): python3-urllib3-1.26.5-4.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2023-12-13T23:53:10.595 INFO:teuthology.orchestra.run.smithi203.stdout:(134/155): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2023-12-13T23:53:10.612 INFO:teuthology.orchestra.run.smithi203.stdout:(135/155): python3-websocket-client-1.2.3-2.el9 566 kB/s | 90 kB 00:00 2023-12-13T23:53:10.663 INFO:teuthology.orchestra.run.smithi203.stdout:(136/155): python3-werkzeug-2.0.3-3.el9.1.noarc 6.3 MB/s | 427 kB 00:00 2023-12-13T23:53:10.688 INFO:teuthology.orchestra.run.smithi203.stdout:(137/155): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2023-12-13T23:53:10.757 INFO:teuthology.orchestra.run.smithi124.stdout:(97/155): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2023-12-13T23:53:10.789 INFO:teuthology.orchestra.run.smithi203.stdout:(138/155): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2023-12-13T23:53:10.830 INFO:teuthology.orchestra.run.smithi203.stdout:(139/155): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-12-13T23:53:11.139 INFO:teuthology.orchestra.run.smithi203.stdout:(140/155): librbd1-19.0.0-157.g021ac167.el9.x86 9.8 MB/s | 3.0 MB 00:00 2023-12-13T23:53:11.172 INFO:teuthology.orchestra.run.smithi203.stdout:(141/155): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-12-13T23:53:11.215 INFO:teuthology.orchestra.run.smithi203.stdout:(142/155): acl-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 69 kB 00:00 2023-12-13T23:53:11.246 INFO:teuthology.orchestra.run.smithi124.stdout:(98/155): gperftools-libs-2.9.1-2.el9.x86_64.rp 457 kB/s | 309 kB 00:00 2023-12-13T23:53:11.248 INFO:teuthology.orchestra.run.smithi203.stdout:(143/155): elfutils-debuginfod-client-0.190-1.e 1.1 MB/s | 36 kB 00:00 2023-12-13T23:53:11.282 INFO:teuthology.orchestra.run.smithi203.stdout:(144/155): elfutils-libelf-0.190-1.el9.x86_64.r 5.7 MB/s | 192 kB 00:00 2023-12-13T23:53:11.316 INFO:teuthology.orchestra.run.smithi203.stdout:(145/155): elfutils-libs-0.190-1.el9.x86_64.rpm 7.5 MB/s | 255 kB 00:00 2023-12-13T23:53:11.349 INFO:teuthology.orchestra.run.smithi203.stdout:(146/155): libacl-2.3.1-4.el9.x86_64.rpm 658 kB/s | 22 kB 00:00 2023-12-13T23:53:11.373 INFO:teuthology.orchestra.run.smithi124.stdout:(99/155): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2023-12-13T23:53:11.391 INFO:teuthology.orchestra.run.smithi203.stdout:(147/155): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2023-12-13T23:53:11.497 INFO:teuthology.orchestra.run.smithi124.stdout:(100/155): liboath-2.6.7-2.el9.x86_64.rpm 389 kB/s | 49 kB 00:00 2023-12-13T23:53:11.614 INFO:teuthology.orchestra.run.smithi124.stdout:(101/155): libunwind-1.6.2-1.el9.x86_64.rpm 578 kB/s | 67 kB 00:00 2023-12-13T23:53:11.700 INFO:teuthology.orchestra.run.smithi203.stdout:(148/155): librados2-19.0.0-157.g021ac167.el9.x 3.6 MB/s | 3.3 MB 00:00 2023-12-13T23:53:11.756 INFO:teuthology.orchestra.run.smithi124.stdout:(102/155): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:01 2023-12-13T23:53:11.781 INFO:teuthology.orchestra.run.smithi124.stdout:(103/155): luarocks-3.9.2-1.el9.noarch.rpm 905 kB/s | 151 kB 00:00 2023-12-13T23:53:11.890 INFO:teuthology.orchestra.run.smithi124.stdout:(104/155): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2023-12-13T23:53:12.032 INFO:teuthology.orchestra.run.smithi124.stdout:(105/155): openssl-devel-3.0.7-25.el9.x86_64.rp 2.7 MB/s | 4.1 MB 00:01 2023-12-13T23:53:12.057 INFO:teuthology.orchestra.run.smithi124.stdout:(106/155): python3-bcrypt-3.2.2-1.el9.x86_64.rp 260 kB/s | 43 kB 00:00 2023-12-13T23:53:12.082 INFO:teuthology.orchestra.run.smithi124.stdout:(107/155): python3-asyncssh-2.13.2-3.el9.noarch 1.8 MB/s | 548 kB 00:00 2023-12-13T23:53:12.183 INFO:teuthology.orchestra.run.smithi124.stdout:(108/155): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2023-12-13T23:53:12.208 INFO:teuthology.orchestra.run.smithi124.stdout:(109/155): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2023-12-13T23:53:12.225 INFO:teuthology.orchestra.run.smithi203.stdout:(149/155): NetworkManager-1.45.7-1.el9.x86_64.r 2.7 MB/s | 2.3 MB 00:00 2023-12-13T23:53:12.284 INFO:teuthology.orchestra.run.smithi203.stdout:(150/155): NetworkManager-team-1.45.7-1.el9.x86 744 kB/s | 43 kB 00:00 2023-12-13T23:53:12.316 INFO:teuthology.orchestra.run.smithi124.stdout:(110/155): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2023-12-13T23:53:12.358 INFO:teuthology.orchestra.run.smithi124.stdout:(111/155): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.0 MB/s | 303 kB 00:00 2023-12-13T23:53:12.418 INFO:teuthology.orchestra.run.smithi203.stdout:(151/155): NetworkManager-tui-1.45.7-1.el9.x86_ 1.8 MB/s | 246 kB 00:00 2023-12-13T23:53:12.434 INFO:teuthology.orchestra.run.smithi124.stdout:(112/155): python3-colorama-0.4.4-12.el9.noarch 301 kB/s | 35 kB 00:00 2023-12-13T23:53:12.450 INFO:teuthology.orchestra.run.smithi124.stdout:(113/155): python3-cachetools-4.2.4-1.el9.noarc 77 kB/s | 32 kB 00:00 2023-12-13T23:53:12.467 INFO:teuthology.orchestra.run.smithi124.stdout:(114/155): python3-google-auth-2.23.4-1.el9.noa 1.8 MB/s | 197 kB 00:00 2023-12-13T23:53:12.543 INFO:teuthology.orchestra.run.smithi124.stdout:(115/155): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2023-12-13T23:53:12.559 INFO:teuthology.orchestra.run.smithi124.stdout:(116/155): python3-jaraco-collections-3.0.0-8.e 252 kB/s | 23 kB 00:00 2023-12-13T23:53:12.576 INFO:teuthology.orchestra.run.smithi203.stdout:(152/155): NetworkManager-libnm-1.45.7-1.el9.x8 2.0 MB/s | 1.8 MB 00:00 2023-12-13T23:53:12.577 INFO:teuthology.orchestra.run.smithi124.stdout:(117/155): python3-jaraco-classes-3.2.1-5.el9.n 140 kB/s | 18 kB 00:00 2023-12-13T23:53:12.635 INFO:teuthology.orchestra.run.smithi124.stdout:(118/155): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2023-12-13T23:53:12.652 INFO:teuthology.orchestra.run.smithi124.stdout:(119/155): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2023-12-13T23:53:12.677 INFO:teuthology.orchestra.run.smithi124.stdout:(120/155): python3-jwt+crypto-2.4.0-1.el9.noarc 90 kB/s | 9.0 kB 00:00 2023-12-13T23:53:12.736 INFO:teuthology.orchestra.run.smithi124.stdout:(121/155): python3-jwt-2.4.0-1.el9.noarch.rpm 407 kB/s | 41 kB 00:00 2023-12-13T23:53:12.786 INFO:teuthology.orchestra.run.smithi124.stdout:(122/155): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2023-12-13T23:53:12.844 INFO:teuthology.orchestra.run.smithi124.stdout:(123/155): python3-logutils-0.3.5-21.el9.noarch 277 kB/s | 46 kB 00:00 2023-12-13T23:53:12.870 INFO:teuthology.orchestra.run.smithi124.stdout:(124/155): python3-more-itertools-8.12.0-2.el9. 589 kB/s | 79 kB 00:00 2023-12-13T23:53:12.903 INFO:teuthology.orchestra.run.smithi124.stdout:(125/155): python3-natsort-7.1.1-5.el9.noarch.r 493 kB/s | 58 kB 00:00 2023-12-13T23:53:12.987 INFO:teuthology.orchestra.run.smithi124.stdout:(126/155): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2023-12-13T23:53:13.012 INFO:teuthology.orchestra.run.smithi124.stdout:(127/155): python3-pyOpenSSL-21.0.0-1.el9.noarc 828 kB/s | 90 kB 00:00 2023-12-13T23:53:13.112 INFO:teuthology.orchestra.run.smithi124.stdout:(128/155): python3-pyperclip-1.8.2-2.el9.noarch 225 kB/s | 28 kB 00:00 2023-12-13T23:53:13.138 INFO:teuthology.orchestra.run.smithi124.stdout:(129/155): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2023-12-13T23:53:13.163 INFO:teuthology.orchestra.run.smithi124.stdout:(130/155): python3-pecan-1.4.2-3.el9.noarch.rpm 856 kB/s | 272 kB 00:00 2023-12-13T23:53:13.230 INFO:teuthology.orchestra.run.smithi124.stdout:(131/155): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2023-12-13T23:53:13.255 INFO:teuthology.orchestra.run.smithi124.stdout:(132/155): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2023-12-13T23:53:13.268 INFO:teuthology.orchestra.run.smithi203.stdout:(153/155): ceph-test-19.0.0-157.g021ac167.el9.x 3.9 MB/s | 41 MB 00:10 2023-12-13T23:53:13.280 INFO:teuthology.orchestra.run.smithi124.stdout:(133/155): python3-tempora-5.0.0-2.el9.noarch.r 305 kB/s | 36 kB 00:00 2023-12-13T23:53:13.302 INFO:teuthology.orchestra.run.smithi203.stdout:(154/155): openssl-3.0.7-25.el9.x86_64.rpm 1.4 MB/s | 1.2 MB 00:00 2023-12-13T23:53:13.347 INFO:teuthology.orchestra.run.smithi124.stdout:(134/155): python3-typing-extensions-4.4.0-2.el 436 kB/s | 51 kB 00:00 2023-12-13T23:53:13.372 INFO:teuthology.orchestra.run.smithi124.stdout:(135/155): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2023-12-13T23:53:13.406 INFO:teuthology.orchestra.run.smithi124.stdout:(136/155): python3-websocket-client-1.2.3-2.el9 715 kB/s | 90 kB 00:00 2023-12-13T23:53:13.473 INFO:teuthology.orchestra.run.smithi124.stdout:(137/155): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2023-12-13T23:53:13.498 INFO:teuthology.orchestra.run.smithi124.stdout:(138/155): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2023-12-13T23:53:13.549 INFO:teuthology.orchestra.run.smithi124.stdout:(139/155): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2023-12-13T23:53:13.602 INFO:teuthology.orchestra.run.smithi203.stdout:(155/155): openssl-libs-3.0.7-25.el9.x86_64.rpm 2.1 MB/s | 2.2 MB 00:01 2023-12-13T23:53:13.608 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2023-12-13T23:53:13.608 INFO:teuthology.orchestra.run.smithi203.stdout:Total 15 MB/s | 205 MB 00:13 2023-12-13T23:53:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:(140/155): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2023-12-13T23:53:13.657 INFO:teuthology.orchestra.run.smithi124.stdout:(141/155): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2023-12-13T23:53:13.691 INFO:teuthology.orchestra.run.smithi124.stdout:(142/155): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2023-12-13T23:53:13.725 INFO:teuthology.orchestra.run.smithi124.stdout:(143/155): elfutils-debuginfod-client-0.190-1.e 1.1 MB/s | 36 kB 00:00 2023-12-13T23:53:13.758 INFO:teuthology.orchestra.run.smithi124.stdout:(144/155): elfutils-libelf-0.190-1.el9.x86_64.r 5.7 MB/s | 192 kB 00:00 2023-12-13T23:53:13.774 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2023-12-13T23:53:13.792 INFO:teuthology.orchestra.run.smithi124.stdout:(145/155): elfutils-libs-0.190-1.el9.x86_64.rpm 7.5 MB/s | 255 kB 00:00 2023-12-13T23:53:13.825 INFO:teuthology.orchestra.run.smithi124.stdout:(146/155): libacl-2.3.1-4.el9.x86_64.rpm 663 kB/s | 22 kB 00:00 2023-12-13T23:53:13.867 INFO:teuthology.orchestra.run.smithi124.stdout:(147/155): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2023-12-13T23:53:13.942 INFO:teuthology.orchestra.run.smithi124.stdout:(148/155): librados2-19.0.0-157.g021ac167.el9.x 7.5 MB/s | 3.3 MB 00:00 2023-12-13T23:53:13.963 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2023-12-13T23:53:13.963 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2023-12-13T23:53:14.044 INFO:teuthology.orchestra.run.smithi124.stdout:(149/155): librbd1-19.0.0-157.g021ac167.el9.x86 6.1 MB/s | 3.0 MB 00:00 2023-12-13T23:53:14.210 INFO:teuthology.orchestra.run.smithi124.stdout:(150/155): NetworkManager-team-1.45.7-1.el9.x86 259 kB/s | 43 kB 00:00 2023-12-13T23:53:14.393 INFO:teuthology.orchestra.run.smithi124.stdout:(151/155): NetworkManager-tui-1.45.7-1.el9.x86_ 1.3 MB/s | 246 kB 00:00 2023-12-13T23:53:14.460 INFO:teuthology.orchestra.run.smithi124.stdout:(152/155): NetworkManager-libnm-1.45.7-1.el9.x8 3.4 MB/s | 1.8 MB 00:00 2023-12-13T23:53:14.594 INFO:teuthology.orchestra.run.smithi124.stdout:(153/155): openssl-3.0.7-25.el9.x86_64.rpm 6.1 MB/s | 1.2 MB 00:00 2023-12-13T23:53:14.636 INFO:teuthology.orchestra.run.smithi124.stdout:(154/155): NetworkManager-1.45.7-1.el9.x86_64.r 2.9 MB/s | 2.3 MB 00:00 2023-12-13T23:53:14.911 INFO:teuthology.orchestra.run.smithi124.stdout:(155/155): openssl-libs-3.0.7-25.el9.x86_64.rpm 4.8 MB/s | 2.2 MB 00:00 2023-12-13T23:53:14.918 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-12-13T23:53:14.918 INFO:teuthology.orchestra.run.smithi124.stdout:Total 15 MB/s | 205 MB 00:13 2023-12-13T23:53:15.088 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-12-13T23:53:15.278 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-12-13T23:53:15.278 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-12-13T23:53:15.787 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2023-12-13T23:53:15.787 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2023-12-13T23:53:17.114 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-12-13T23:53:17.114 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-12-13T23:53:21.889 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2023-12-13T23:53:22.023 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/170 2023-12-13T23:53:22.065 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/170 2023-12-13T23:53:22.104 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/170 2023-12-13T23:53:22.148 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/170 2023-12-13T23:53:22.174 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2023-12-13T23:53:22.230 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2023-12-13T23:53:22.552 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2023-12-13T23:53:22.664 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:53:23.040 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:53:23.068 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:53:23.158 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:53:23.194 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 10/170 2023-12-13T23:53:23.244 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/170 2023-12-13T23:53:23.284 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-12-13T23:53:23.412 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/170 2023-12-13T23:53:23.416 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/170 2023-12-13T23:53:23.440 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/170 2023-12-13T23:53:23.463 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/170 2023-12-13T23:53:23.488 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/170 2023-12-13T23:53:23.491 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:53:23.531 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/170 2023-12-13T23:53:23.557 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2023-12-13T23:53:23.575 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:53:23.600 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 15/170 2023-12-13T23:53:23.606 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2023-12-13T23:53:23.789 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 15/170 2023-12-13T23:53:23.922 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2023-12-13T23:53:23.929 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-13T23:53:23.983 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librados2-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:53:23.989 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-13T23:53:24.090 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:24.101 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:53:24.126 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:53:24.149 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:24.217 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:53:24.252 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 10/170 2023-12-13T23:53:24.299 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/170 2023-12-13T23:53:24.356 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:24.452 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/170 2023-12-13T23:53:24.475 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : elfutils-libelf-0.190-1.el9.x86_64 12/170 2023-12-13T23:53:24.487 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/170 2023-12-13T23:53:24.527 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-13.11-1.el9.x86_64 13/170 2023-12-13T23:53:24.555 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:53:24.558 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/170 2023-12-13T23:53:24.610 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/170 2023-12-13T23:53:24.641 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/170 2023-12-13T23:53:24.657 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:53:24.681 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 15/170 2023-12-13T23:53:24.686 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/170 2023-12-13T23:53:24.736 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/170 2023-12-13T23:53:24.816 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/170 2023-12-13T23:53:24.850 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/170 2023-12-13T23:53:24.877 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/170 2023-12-13T23:53:24.878 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 15/170 2023-12-13T23:53:24.948 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2023-12-13T23:53:24.982 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x 29/170 2023-12-13T23:53:24.993 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-13T23:53:25.026 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9 30/170 2023-12-13T23:53:25.053 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 16/170 2023-12-13T23:53:25.098 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 31/170 2023-12-13T23:53:25.134 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:25.187 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:25.240 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/170 2023-12-13T23:53:25.303 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/170 2023-12-13T23:53:25.404 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-1:1.45.7-1.el9.x86_64 17/170 2023-12-13T23:53:25.505 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 18/170 2023-12-13T23:53:25.542 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/170 2023-12-13T23:53:25.619 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/170 2023-12-13T23:53:25.673 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 21/170 2023-12-13T23:53:25.703 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 22/170 2023-12-13T23:53:25.748 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 23/170 2023-12-13T23:53:25.787 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 24/170 2023-12-13T23:53:25.825 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/170 2023-12-13T23:53:25.860 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/170 2023-12-13T23:53:25.894 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/170 2023-12-13T23:53:25.906 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/170 2023-12-13T23:53:25.922 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/170 2023-12-13T23:53:25.946 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/170 2023-12-13T23:53:25.988 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/170 2023-12-13T23:53:25.992 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2023-12-13T23:53:26.026 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 38/170 2023-12-13T23:53:26.026 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x 29/170 2023-12-13T23:53:26.068 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/170 2023-12-13T23:53:26.070 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9 30/170 2023-12-13T23:53:26.144 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 31/170 2023-12-13T23:53:26.217 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/170 2023-12-13T23:53:26.260 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/170 2023-12-13T23:53:26.288 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 32/170 2023-12-13T23:53:26.339 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 33/170 2023-12-13T23:53:26.368 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/170 2023-12-13T23:53:26.405 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 43/170 2023-12-13T23:53:26.460 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/170 2023-12-13T23:53:26.525 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/170 2023-12-13T23:53:26.565 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/170 2023-12-13T23:53:26.601 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/170 2023-12-13T23:53:26.648 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/170 2023-12-13T23:53:26.684 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/170 2023-12-13T23:53:26.727 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/170 2023-12-13T23:53:26.769 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/170 2023-12-13T23:53:26.893 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rsa-4.9-2.el9.noarch 34/170 2023-12-13T23:53:26.902 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/170 2023-12-13T23:53:26.950 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/170 2023-12-13T23:53:26.976 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 35/170 2023-12-13T23:53:26.980 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/170 2023-12-13T23:53:27.011 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 36/170 2023-12-13T23:53:27.022 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/170 2023-12-13T23:53:27.064 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : elfutils-libs-0.190-1.el9.x86_64 37/170 2023-12-13T23:53:27.102 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 38/170 2023-12-13T23:53:27.116 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/170 2023-12-13T23:53:27.144 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 39/170 2023-12-13T23:53:27.149 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 57/170 2023-12-13T23:53:27.189 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/170 2023-12-13T23:53:27.222 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/170 2023-12-13T23:53:27.283 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 40/170 2023-12-13T23:53:27.316 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/170 2023-12-13T23:53:27.336 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 41/170 2023-12-13T23:53:27.383 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/170 2023-12-13T23:53:27.441 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/170 2023-12-13T23:53:27.442 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : re2-1:20211101-3.el9.x86_64 42/170 2023-12-13T23:53:27.481 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 43/170 2023-12-13T23:53:27.497 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/170 2023-12-13T23:53:27.536 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 44/170 2023-12-13T23:53:27.609 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 45/170 2023-12-13T23:53:27.639 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 46/170 2023-12-13T23:53:27.684 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 47/170 2023-12-13T23:53:27.686 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/170 2023-12-13T23:53:27.731 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 48/170 2023-12-13T23:53:27.738 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/170 2023-12-13T23:53:27.767 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 49/170 2023-12-13T23:53:27.810 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 50/170 2023-12-13T23:53:27.852 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 51/170 2023-12-13T23:53:27.876 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/170 2023-12-13T23:53:27.915 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/170 2023-12-13T23:53:27.945 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/170 2023-12-13T23:53:27.979 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/170 2023-12-13T23:53:27.986 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 52/170 2023-12-13T23:53:28.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/170 2023-12-13T23:53:28.040 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 53/170 2023-12-13T23:53:28.050 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 71/170 2023-12-13T23:53:28.071 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 54/170 2023-12-13T23:53:28.095 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 72/170 2023-12-13T23:53:28.112 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 55/170 2023-12-13T23:53:28.146 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/170 2023-12-13T23:53:28.185 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/170 2023-12-13T23:53:28.205 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 56/170 2023-12-13T23:53:28.217 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/170 2023-12-13T23:53:28.238 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 57/170 2023-12-13T23:53:28.247 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/170 2023-12-13T23:53:28.278 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 58/170 2023-12-13T23:53:28.311 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 59/170 2023-12-13T23:53:28.324 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/170 2023-12-13T23:53:28.358 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/170 2023-12-13T23:53:28.406 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 60/170 2023-12-13T23:53:28.455 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 61/170 2023-12-13T23:53:28.504 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ply-3.11-14.el9.noarch 62/170 2023-12-13T23:53:28.559 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 63/170 2023-12-13T23:53:28.601 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/170 2023-12-13T23:53:28.643 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/170 2023-12-13T23:53:28.677 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/170 2023-12-13T23:53:28.744 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 64/170 2023-12-13T23:53:28.772 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/170 2023-12-13T23:53:28.784 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 65/170 2023-12-13T23:53:28.804 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/170 2023-12-13T23:53:28.909 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/170 2023-12-13T23:53:28.911 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 66/170 2023-12-13T23:53:28.962 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 67/170 2023-12-13T23:53:28.963 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2023-12-13T23:53:28.991 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 68/170 2023-12-13T23:53:29.031 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 69/170 2023-12-13T23:53:29.070 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 70/170 2023-12-13T23:53:29.100 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : fuse-2.9.9-15.el9.x86_64 71/170 2023-12-13T23:53:29.300 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 72/170 2023-12-13T23:53:29.457 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 73/170 2023-12-13T23:53:29.495 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : uuid-1.6.2-55.el9.x86_64 74/170 2023-12-13T23:53:29.535 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 75/170 2023-12-13T23:53:29.564 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 76/170 2023-12-13T23:53:29.644 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 77/170 2023-12-13T23:53:29.674 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 78/170 2023-12-13T23:53:29.700 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/170 2023-12-13T23:53:29.707 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 79/170 2023-12-13T23:53:29.744 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/170 2023-12-13T23:53:29.749 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 80/170 2023-12-13T23:53:29.778 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/170 2023-12-13T23:53:29.783 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 81/170 2023-12-13T23:53:29.828 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/170 2023-12-13T23:53:29.873 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 82/170 2023-12-13T23:53:29.910 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 83/170 2023-12-13T23:53:29.913 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/170 2023-12-13T23:53:29.945 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/170 2023-12-13T23:53:30.012 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 84/170 2023-12-13T23:53:30.061 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2023-12-13T23:53:30.127 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 92/170 2023-12-13T23:53:30.169 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 93/170 2023-12-13T23:53:30.257 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/170 2023-12-13T23:53:30.784 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/170 2023-12-13T23:53:30.833 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/170 2023-12-13T23:53:30.866 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/170 2023-12-13T23:53:30.905 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/170 2023-12-13T23:53:30.915 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/170 2023-12-13T23:53:31.004 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 90/170 2023-12-13T23:53:31.025 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/170 2023-12-13T23:53:31.034 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 91/170 2023-12-13T23:53:31.211 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-0.3.21-2.el9.x86_64 92/170 2023-12-13T23:53:31.231 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 93/170 2023-12-13T23:53:31.310 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 94/170 2023-12-13T23:53:31.964 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 95/170 2023-12-13T23:53:32.162 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 96/170 2023-12-13T23:53:32.266 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/170 2023-12-13T23:53:32.355 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/170 2023-12-13T23:53:32.393 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 99/170 2023-12-13T23:53:32.423 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 100/170 2023-12-13T23:53:32.494 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 101/170 2023-12-13T23:53:32.527 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 102/170 2023-12-13T23:53:32.556 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 103/170 2023-12-13T23:53:32.590 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 104/170 2023-12-13T23:53:32.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/170 2023-12-13T23:53:32.920 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 106/170 2023-12-13T23:53:32.993 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 106/170 2023-12-13T23:53:33.024 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 107/170 2023-12-13T23:53:33.061 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/170 2023-12-13T23:53:33.409 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 97/170 2023-12-13T23:53:33.486 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 109/170 2023-12-13T23:53:33.497 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 98/170 2023-12-13T23:53:33.522 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 110/170 2023-12-13T23:53:33.529 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 99/170 2023-12-13T23:53:33.565 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : jq-1.6-16.el9.x86_64 100/170 2023-12-13T23:53:33.593 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 110/170 2023-12-13T23:53:33.616 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 111/170 2023-12-13T23:53:33.629 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lua-5.4.4-4.el9.x86_64 101/170 2023-12-13T23:53:33.662 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : luarocks-3.9.2-1.el9.noarch 102/170 2023-12-13T23:53:33.690 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 103/170 2023-12-13T23:53:33.723 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 104/170 2023-12-13T23:53:34.024 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 105/170 2023-12-13T23:53:34.053 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 106/170 2023-12-13T23:53:34.134 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 106/170 2023-12-13T23:53:34.165 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 107/170 2023-12-13T23:53:34.201 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 108/170 2023-12-13T23:53:34.624 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 109/170 2023-12-13T23:53:34.647 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 110/170 2023-12-13T23:53:34.718 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 110/170 2023-12-13T23:53:34.740 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 111/170 2023-12-13T23:53:35.648 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:35.688 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:36.224 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:36.259 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 113/170 2023-12-13T23:53:36.339 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 113/170 2023-12-13T23:53:36.339 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-13T23:53:36.339 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-13T23:53:36.340 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:36.377 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 114/170 2023-12-13T23:53:36.847 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:36.878 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:37.415 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 112/170 2023-12-13T23:53:37.464 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 113/170 2023-12-13T23:53:37.544 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 113/170 2023-12-13T23:53:37.544 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-13T23:53:37.544 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-13T23:53:37.544 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:37.574 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 114/170 2023-12-13T23:53:48.386 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 114/170 2023-12-13T23:53:48.386 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2023-12-13T23:53:48.386 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2023-12-13T23:53:48.386 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2023-12-13T23:53:48.386 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2023-12-13T23:53:48.387 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2023-12-13T23:53:48.387 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2023-12-13T23:53:48.387 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2023-12-13T23:53:48.387 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:48.545 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 115/170 2023-12-13T23:53:48.560 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 115/170 2023-12-13T23:53:48.560 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:48.560 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-13T23:53:48.561 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:49.014 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 114/170 2023-12-13T23:53:49.014 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /sys 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /proc 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /mnt 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /var/tmp 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /home 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /root 2023-12-13T23:53:49.015 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /tmp 2023-12-13T23:53:49.016 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:49.184 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 115/170 2023-12-13T23:53:49.199 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 115/170 2023-12-13T23:53:49.199 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:49.200 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-13T23:53:49.200 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:50.131 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 116/170 2023-12-13T23:53:50.156 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 116/170 2023-12-13T23:53:50.157 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:50.157 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-13T23:53:50.157 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-13T23:53:50.157 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-13T23:53:50.157 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:50.366 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 117/170 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 117/170 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-13T23:53:50.389 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:50.753 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 116/170 2023-12-13T23:53:50.779 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 116/170 2023-12-13T23:53:50.779 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:50.779 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-13T23:53:50.779 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-13T23:53:50.780 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-13T23:53:50.780 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:50.843 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 118/170 2023-12-13T23:53:50.866 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 118/170 2023-12-13T23:53:50.866 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:50.866 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-13T23:53:50.866 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-13T23:53:50.866 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-13T23:53:50.867 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:50.962 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : automake-1.16.2-8.el9.noarch 119/170 2023-12-13T23:53:50.979 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 117/170 2023-12-13T23:53:51.001 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 117/170 2023-12-13T23:53:51.001 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:51.002 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-13T23:53:51.002 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-13T23:53:51.002 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-13T23:53:51.002 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:51.398 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 120/170 2023-12-13T23:53:51.435 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-157.g021ac167.el9.noarch 120/170 2023-12-13T23:53:51.455 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 118/170 2023-12-13T23:53:51.465 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el 121/170 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 118/170 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-13T23:53:51.478 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:51.526 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.e 122/170 2023-12-13T23:53:51.550 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 123/170 2023-12-13T23:53:51.564 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : automake-1.16.2-8.el9.noarch 119/170 2023-12-13T23:53:51.776 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 123/170 2023-12-13T23:53:51.818 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 124/170 2023-12-13T23:53:51.967 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephadm-2:19.0.0-157.g021ac167.el9.noarch 120/170 2023-12-13T23:53:52.004 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephadm-2:19.0.0-157.g021ac167.el9.noarch 120/170 2023-12-13T23:53:52.034 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el 121/170 2023-12-13T23:53:52.094 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.e 122/170 2023-12-13T23:53:52.127 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 123/170 2023-12-13T23:53:52.351 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 123/170 2023-12-13T23:53:52.378 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 124/170 2023-12-13T23:53:52.778 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 124/170 2023-12-13T23:53:52.802 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 125/170 2023-12-13T23:53:52.900 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 125/170 2023-12-13T23:53:53.059 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9 126/170 2023-12-13T23:53:53.096 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 127/170 2023-12-13T23:53:53.118 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 127/170 2023-12-13T23:53:53.119 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:53.119 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-13T23:53:53.119 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-13T23:53:53.119 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-13T23:53:53.119 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:53.163 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 128/170 2023-12-13T23:53:53.174 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 128/170 2023-12-13T23:53:53.228 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-157.g021ac167.el9.x86_64 129/170 2023-12-13T23:53:53.338 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 124/170 2023-12-13T23:53:53.362 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 125/170 2023-12-13T23:53:53.463 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 125/170 2023-12-13T23:53:53.618 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9 126/170 2023-12-13T23:53:53.647 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 127/170 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 127/170 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-13T23:53:53.671 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:53.722 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 128/170 2023-12-13T23:53:53.733 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 128/170 2023-12-13T23:53:53.786 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-2:19.0.0-157.g021ac167.el9.x86_64 129/170 2023-12-13T23:53:53.995 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/170 2023-12-13T23:53:54.024 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 131/170 2023-12-13T23:53:54.046 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 131/170 2023-12-13T23:53:54.047 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.047 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-13T23:53:54.047 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-13T23:53:54.047 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-13T23:53:54.047 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:54.083 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-157.g021ac1 132/170 2023-12-13T23:53:54.105 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-157.g021ac1 132/170 2023-12-13T23:53:54.105 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.105 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-13T23:53:54.105 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:54.158 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 133/170 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 133/170 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-13T23:53:54.181 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:54.476 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 134/170 2023-12-13T23:53:54.498 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 134/170 2023-12-13T23:53:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-13T23:53:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-13T23:53:54.499 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-13T23:53:54.499 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:53:54.557 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/170 2023-12-13T23:53:54.583 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 131/170 2023-12-13T23:53:54.605 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 131/170 2023-12-13T23:53:54.605 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.605 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-13T23:53:54.606 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-13T23:53:54.606 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-13T23:53:54.606 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:54.684 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-157.g021ac1 132/170 2023-12-13T23:53:54.706 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-157.g021ac1 132/170 2023-12-13T23:53:54.706 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.706 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-13T23:53:54.706 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:54.751 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 133/170 2023-12-13T23:53:54.773 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 133/170 2023-12-13T23:53:54.774 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:54.774 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-12-13T23:53:54.774 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-13T23:53:54.777 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-12-13T23:53:54.777 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:55.077 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 134/170 2023-12-13T23:53:55.099 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 134/170 2023-12-13T23:53:55.099 INFO:teuthology.orchestra.run.smithi124.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-13T23:53:55.099 INFO:teuthology.orchestra.run.smithi124.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-13T23:53:55.100 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-13T23:53:55.100 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-13T23:53:55.100 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:53:58.330 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 135/170 2023-12-13T23:53:58.371 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 136/170 2023-12-13T23:53:58.412 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 137/170 2023-12-13T23:53:58.444 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 138/170 2023-12-13T23:53:58.471 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 139/170 2023-12-13T23:53:58.653 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 140/170 2023-12-13T23:53:58.760 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 141/170 2023-12-13T23:53:58.794 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 142/170 2023-12-13T23:53:58.827 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_6 143/170 2023-12-13T23:53:58.891 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/170 2023-12-13T23:53:58.921 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 135/170 2023-12-13T23:53:58.950 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-13T23:53:58.962 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 136/170 2023-12-13T23:53:58.978 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 146/170 2023-12-13T23:53:59.003 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 137/170 2023-12-13T23:53:59.008 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 147/170 2023-12-13T23:53:59.035 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 138/170 2023-12-13T23:53:59.054 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 148/170 2023-12-13T23:53:59.062 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : elfutils-libelf-devel-0.190-1.el9.x86_64 139/170 2023-12-13T23:53:59.255 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 140/170 2023-12-13T23:53:59.369 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 141/170 2023-12-13T23:53:59.402 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 142/170 2023-12-13T23:53:59.435 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_6 143/170 2023-12-13T23:53:59.495 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : elfutils-debuginfod-client-0.190-1.el9.x86_64 144/170 2023-12-13T23:53:59.548 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-13T23:53:59.577 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 146/170 2023-12-13T23:53:59.599 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : NetworkManager-team-1:1.45.7-1.el9.x86_64 147/170 2023-12-13T23:53:59.636 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : NetworkManager-tui-1:1.45.7-1.el9.x86_64 148/170 2023-12-13T23:53:59.681 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:53:59.721 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:53:59.740 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:54:00.264 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:54:00.301 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:54:00.322 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 149/170 2023-12-13T23:54:00.561 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 150/170 2023-12-13T23:54:00.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 151/170 2023-12-13T23:54:00.726 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 152/170 2023-12-13T23:54:00.767 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 153/170 2023-12-13T23:54:00.849 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flex-2.6.4-9.el9.x86_64 154/170 2023-12-13T23:54:00.881 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : bison-3.7.4-5.el9.x86_64 155/170 2023-12-13T23:54:00.906 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 156/170 2023-12-13T23:54:00.931 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 157/170 2023-12-13T23:54:00.958 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 158/170 2023-12-13T23:54:00.958 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-13T23:54:01.043 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-13T23:54:01.070 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 160/170 2023-12-13T23:54:01.105 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/170 2023-12-13T23:54:01.126 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 162/170 2023-12-13T23:54:01.126 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-13T23:54:01.140 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 150/170 2023-12-13T23:54:01.219 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-13T23:54:01.244 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-13T23:54:01.259 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:01.259 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:01.266 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 151/170 2023-12-13T23:54:01.313 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 152/170 2023-12-13T23:54:01.356 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 153/170 2023-12-13T23:54:01.439 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:01.439 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:01.447 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flex-2.6.4-9.el9.x86_64 154/170 2023-12-13T23:54:01.475 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : bison-3.7.4-5.el9.x86_64 155/170 2023-12-13T23:54:01.500 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 156/170 2023-12-13T23:54:01.515 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:01.526 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 157/170 2023-12-13T23:54:01.539 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 167/170 2023-12-13T23:54:01.552 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 158/170 2023-12-13T23:54:01.552 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-13T23:54:01.564 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 168/170 2023-12-13T23:54:01.590 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 169/170 2023-12-13T23:54:01.637 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 159/170 2023-12-13T23:54:01.664 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-13T23:54:01.664 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 160/170 2023-12-13T23:54:01.704 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/170 2023-12-13T23:54:01.729 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 162/170 2023-12-13T23:54:01.730 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-13T23:54:01.813 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 163/170 2023-12-13T23:54:01.838 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-13T23:54:01.844 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:01.845 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:02.023 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 165/170 2023-12-13T23:54:02.023 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:02.091 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:02.115 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 167/170 2023-12-13T23:54:02.140 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 168/170 2023-12-13T23:54:02.165 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 169/170 2023-12-13T23:54:02.231 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-157.g021ac167.el9.x86_64 1/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 3/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 4/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-157.g021ac1 5/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 6/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 7/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:54:06.395 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 10/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 11/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 12/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 13/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_6 15/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 16/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 17/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 18/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 19/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 20/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9 21/170 2023-12-13T23:54:06.396 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x 22/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 23/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 24/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 25/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 26/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 27/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 28/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 29/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.e 30/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 31/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 32/170 2023-12-13T23:54:06.397 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 33/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9 34/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 35/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el 36/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 37/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-157.g021ac167.el9.noarch 38/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 39/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/170 2023-12-13T23:54:06.398 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 43/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 48/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 49/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 53/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 54/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 55/170 2023-12-13T23:54:06.399 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 58/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 59/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 60/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-13.11-1.el9.x86_64 61/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 62/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 63/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 64/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 67/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/170 2023-12-13T23:54:06.400 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 76/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 77/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 78/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 80/170 2023-12-13T23:54:06.401 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 81/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 82/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 83/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 84/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 85/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 86/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 87/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 88/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 89/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 90/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 91/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 92/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 93/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 94/170 2023-12-13T23:54:06.402 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 95/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 96/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 97/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 98/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 99/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 100/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 101/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 102/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 103/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 104/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 105/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 106/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/170 2023-12-13T23:54:06.403 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 109/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 110/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 112/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 113/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 114/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 118/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 119/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 120/170 2023-12-13T23:54:06.404 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 121/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 122/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 123/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 124/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 125/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 126/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 127/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 128/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 129/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 130/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 131/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 132/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 133/170 2023-12-13T23:54:06.405 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 134/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 135/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 136/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 137/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 138/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 139/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 140/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-157.g021ac167.el9.x86_64 141/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 142/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 143/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 144/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 146/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/170 2023-12-13T23:54:06.406 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/170 2023-12-13T23:54:06.407 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 159/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 160/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 161/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 162/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 163/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 165/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 167/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/170 2023-12-13T23:54:06.408 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 169/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 170/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-2:19.0.0-157.g021ac167.el9.x86_64 1/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 3/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 4/170 2023-12-13T23:54:06.859 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-157.g021ac1 5/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 6/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 7/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 8/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 9/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 10/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 11/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 12/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 13/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 14/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_6 15/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 16/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 17/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 18/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_ 19/170 2023-12-13T23:54:06.860 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librgw2-2:19.0.0-157.g021ac167.el9.x86_64 20/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-argparse-2:19.0.0-157.g021ac167.el9 21/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-common-2:19.0.0-157.g021ac167.el9.x 22/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 23/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 24/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 25/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 26/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 27/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 28/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 29/170 2023-12-13T23:54:06.861 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-157.g021ac167.e 30/170 2023-12-13T23:54:06.875 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noar 31/170 2023-12-13T23:54:06.875 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.no 32/170 2023-12-13T23:54:06.875 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-157.g021a 33/170 2023-12-13T23:54:06.875 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9 34/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 35/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el 36/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 37/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephadm-2:19.0.0-157.g021ac167.el9.noarch 38/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 39/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-devel-0.190-1.el9.x86_64 43/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/170 2023-12-13T23:54:06.876 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : jq-1.6-16.el9.x86_64 48/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 49/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 50/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 51/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 52/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 53/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 54/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 55/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 56/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lua-5.4.4-4.el9.x86_64 57/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 58/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 59/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 60/170 2023-12-13T23:54:06.877 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-13.11-1.el9.x86_64 61/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 62/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 63/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 64/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 67/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/170 2023-12-13T23:54:06.878 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 75/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 76/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 77/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 78/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 79/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 80/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 81/170 2023-12-13T23:54:06.879 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 82/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 83/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 84/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 85/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 86/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 87/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 88/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 89/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 90/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 91/170 2023-12-13T23:54:06.880 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 92/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ply-3.11-14.el9.noarch 93/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 94/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 95/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 96/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 97/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 98/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 99/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 100/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 101/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 102/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 103/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 104/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 105/170 2023-12-13T23:54:06.881 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 106/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 107/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 108/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 109/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 110/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 111/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 112/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 113/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 114/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 115/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 116/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 117/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 118/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 119/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 120/170 2023-12-13T23:54:06.882 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 121/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 122/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 123/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 124/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 125/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 126/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 127/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 128/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 129/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 130/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 131/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 132/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 133/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 134/170 2023-12-13T23:54:06.883 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 135/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 136/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 137/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 138/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : re2-1:20211101-3.el9.x86_64 139/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 140/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-2:19.0.0-157.g021ac167.el9.x86_64 141/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 142/170 2023-12-13T23:54:06.884 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-2:19.0.0-157.g021ac167.el9.x86_64 143/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 144/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 145/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 146/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-debuginfod-client-0.190-1.el9.x86_64 149/170 2023-12-13T23:54:06.885 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-0.190-1.el9.x86_64 151/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libs-0.190-1.el9.x86_64 153/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/170 2023-12-13T23:54:06.886 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 157/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 158/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-1:1.45.7-1.el9.x86_64 159/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 160/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-libnm-1:1.45.7-1.el9.x86_64 161/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 162/170 2023-12-13T23:54:06.887 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-team-1:1.45.7-1.el9.x86_64 163/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 164/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-tui-1:1.45.7-1.el9.x86_64 165/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 166/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 167/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/170 2023-12-13T23:54:06.888 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 169/170 2023-12-13T23:54:08.976 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/170 2023-12-13T23:54:08.976 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:08.976 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2023-12-13T23:54:08.976 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:08.976 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: acl-2.3.1-4.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-2.3.1-4.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.977 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: automake-1.16.2-8.el9.noarch 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: bison-3.7.4-5.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.978 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.979 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: flex-2.6.4-9.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-13T23:54:08.980 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-13T23:54:08.981 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libtool-2.4.6-45.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-12-13T23:54:08.982 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-13.11-1.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: postgresql-server-13.11-1.el9.x86_64 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-13T23:54:08.983 INFO:teuthology.orchestra.run.smithi203.stdout: python3-attrs-20.3.0-7.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cmd2-2.4.2-2.el9.noarch 2023-12-13T23:54:08.984 INFO:teuthology.orchestra.run.smithi203.stdout: python3-colorama-0.4.4-12.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-13T23:54:08.985 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-13T23:54:08.986 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.987 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-13T23:54:08.988 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: uuid-1.6.2-55.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-12-13T23:54:08.989 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-13T23:54:08.990 INFO:teuthology.orchestra.run.smithi203.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-12-13T23:54:08.990 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:08.990 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/170 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout:Upgraded: 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-libnm-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:09.252 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-team-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: NetworkManager-tui-1:1.45.7-1.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: acl-2.3.1-4.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-debuginfod-client-0.190-1.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-0.190-1.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libs-0.190-1.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: libacl-2.3.1-4.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: librados2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:09.253 INFO:teuthology.orchestra.run.smithi124.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: automake-1.16.2-8.el9.noarch 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: bison-3.7.4-5.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.254 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.255 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-shell-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top-2:19.0.0-157.g021ac167.el9.noarch 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel-0.190-1.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: flex-2.6.4-9.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: fuse-2.9.9-15.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: jq-1.6-16.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: libarrow-9.0.0-10.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: libarrow-doc-9.0.0-10.el9.noarch 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.256 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libtool-2.4.6-45.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-13T23:54:09.257 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: lua-5.4.4-4.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: parquet-libs-9.0.0-10.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-13.11-1.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: postgresql-server-13.11-1.el9.x86_64 2023-12-13T23:54:09.258 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-attrs-20.3.0-7.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cmd2-2.4.2-2.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-colorama-0.4.4-12.el9.noarch 2023-12-13T23:54:09.259 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-13T23:54:09.260 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ply-3.11-14.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-13T23:54:09.261 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-13T23:54:09.262 INFO:teuthology.orchestra.run.smithi124.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd-2:19.0.0-157.g021ac167.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: re2-1:20211101-3.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: uuid-1.6.2-55.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-12-13T23:54:09.263 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-13T23:54:09.264 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-12-13T23:54:09.264 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:09.264 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-12-13T23:54:09.439 DEBUG:teuthology.parallel:result is None 2023-12-13T23:54:09.801 DEBUG:teuthology.parallel:result is None 2023-12-13T23:54:09.802 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-13T23:54:10.043 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-13T23:54:10.065 INFO:teuthology.orchestra.run.smithi124.stdout:19.0.0-157.g021ac167.el9 2023-12-13T23:54:10.066 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-157.g021ac167.el9 2023-12-13T23:54:10.066 INFO:teuthology.task.install:The correct ceph version 19.0.0-157.g021ac167 is installed. 2023-12-13T23:54:10.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=021ac1670a21c19759080c6da90baaeb42e7d175 2023-12-13T23:54:10.284 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-13T23:54:10.309 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-157.g021ac167.el9 2023-12-13T23:54:10.309 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-157.g021ac167.el9 2023-12-13T23:54:10.309 INFO:teuthology.task.install:The correct ceph version 19.0.0-157.g021ac167 is installed. 2023-12-13T23:54:10.315 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-13T23:54:10.315 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:10.315 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-13T23:54:10.349 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:10.349 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-13T23:54:10.383 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-13T23:54:10.383 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:10.383 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/daemon-helper 2023-12-13T23:54:10.419 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-13T23:54:10.488 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:10.488 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2023-12-13T23:54:10.521 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-13T23:54:10.587 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-13T23:54:10.587 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:10.588 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-13T23:54:10.617 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-13T23:54:10.686 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:10.687 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-13T23:54:10.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-13T23:54:10.783 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-13T23:54:10.783 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:10.783 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/stdin-killer 2023-12-13T23:54:10.812 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-13T23:54:10.880 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:10.880 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2023-12-13T23:54:10.908 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-13T23:54:10.976 INFO:teuthology.run_tasks:Running task ceph... 2023-12-13T23:54:11.045 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-13T23:54:11.045 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 777 /var/log/ceph 2023-12-13T23:54:11.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2023-12-13T23:54:11.075 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-13T23:54:11.075 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-13T23:54:11.117 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-13T23:54:11.144 INFO:tasks.ceph:Creating extra log directories... 2023-12-13T23:54:11.144 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-13T23:54:11.186 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-13T23:54:11.218 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-12-13T23:54:11.218 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-12-13T23:54:11.218 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-12-13T23:54:11.218 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:11.218 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-13T23:54:11.266 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:11.385 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:11.413 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:11.413 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-12-13T23:54:11.428 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:11.544 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:11.576 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-13T23:54:11.576 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-13T23:54:11.577 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669', 'branch': 'wip-yuri10-testing-2023-12-12-1229', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7490669', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri10-testing-2023-12-12-1229'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri10-testing-2023-12-12-1229', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 8548, 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri10-testing-2023-12-12-1229', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi124.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBqrEjhIl+df54aDgFucXunpO4UgObFWJ9ytkz5DhJ26N/0p3E8peMvYr//WC8rqdADbgVMPgTdeyRMyUl8nI8=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJA14Vswc23fItM+D5R1LMCctN4imLdwSpzC1urUzN+ZX/tklb8W2t0fbPe7MwqXCylSmjMNwo57jeZjMMTgPO0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '021ac1670a21c19759080c6da90baaeb42e7d175', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-13_20:38:28', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-13T23:54:11.577 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-13T23:54:11.581 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-13T23:54:11.630 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2023-12-13T23:54:11.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2023-12-13T23:54:11.701 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:11.701 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/scratch_devs of=/dev/stdout 2023-12-13T23:54:11.727 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-13T23:54:11.727 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_1 2023-12-13T23:54:11.784 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-13T23:54:11.784 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:11.784 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 5h/5d Inode: 665 Links: 1 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-12-13 23:54:02.569707430 +0000 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-12-13 23:52:05.439068275 +0000 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-12-13 23:52:05.439068275 +0000 2023-12-13T23:54:11.785 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: 2023-12-13 23:52:05.439068275 +0000 2023-12-13T23:54:11.785 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-13T23:54:11.857 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-12-13T23:54:11.857 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-12-13T23:54:11.857 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000225255 s, 2.3 MB/s 2023-12-13T23:54:11.858 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-13T23:54:11.918 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_2 2023-12-13T23:54:11.977 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-12-13 23:54:02.569707430 +0000 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-12-13 23:52:05.940059251 +0000 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-12-13 23:52:05.940059251 +0000 2023-12-13T23:54:11.978 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: 2023-12-13 23:52:05.940059251 +0000 2023-12-13T23:54:11.978 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-13T23:54:12.044 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-12-13T23:54:12.045 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-12-13T23:54:12.045 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000220048 s, 2.3 MB/s 2023-12-13T23:54:12.046 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-13T23:54:12.102 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_3 2023-12-13T23:54:12.157 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-13T23:54:12.157 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 5h/5d Inode: 703 Links: 1 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-12-13 23:54:02.570707412 +0000 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-12-13 23:52:06.446050136 +0000 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-12-13 23:52:06.446050136 +0000 2023-12-13T23:54:12.158 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: 2023-12-13 23:52:06.446050136 +0000 2023-12-13T23:54:12.158 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-13T23:54:12.222 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-12-13T23:54:12.222 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-12-13T23:54:12.223 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000223394 s, 2.3 MB/s 2023-12-13T23:54:12.224 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-13T23:54:12.279 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_4 2023-12-13T23:54:12.333 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-13T23:54:12.333 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:12.333 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-13T23:54:12.333 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:12.333 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:12.334 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-12-13 23:54:02.570707412 +0000 2023-12-13T23:54:12.334 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-12-13 23:52:06.936041309 +0000 2023-12-13T23:54:12.334 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-12-13 23:52:06.936041309 +0000 2023-12-13T23:54:12.334 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: 2023-12-13 23:52:06.936041309 +0000 2023-12-13T23:54:12.334 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-13T23:54:12.401 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-12-13T23:54:12.402 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-12-13T23:54:12.402 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.00016972 s, 3.0 MB/s 2023-12-13T23:54:12.402 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-13T23:54:12.457 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:12.457 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:12.457 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2023-12-13T23:54:12.474 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-13T23:54:12.474 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2023-12-13T23:54:12.530 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-12-13 23:54:01.975530466 +0000 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-12-13 23:52:05.430609414 +0000 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-12-13 23:52:05.430609414 +0000 2023-12-13T23:54:12.531 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2023-12-13 23:52:05.430609414 +0000 2023-12-13T23:54:12.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-13T23:54:12.599 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-12-13T23:54:12.600 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-12-13T23:54:12.600 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000193974 s, 2.6 MB/s 2023-12-13T23:54:12.601 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-13T23:54:12.658 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2023-12-13T23:54:12.714 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-13T23:54:12.714 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:12.714 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-13T23:54:12.714 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:12.715 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:12.715 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-12-13 23:54:01.976530441 +0000 2023-12-13T23:54:12.715 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-12-13 23:52:05.928596854 +0000 2023-12-13T23:54:12.715 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-12-13 23:52:05.928596854 +0000 2023-12-13T23:54:12.715 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2023-12-13 23:52:05.928596854 +0000 2023-12-13T23:54:12.715 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-13T23:54:12.781 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-12-13T23:54:12.781 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-12-13T23:54:12.781 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000259941 s, 2.0 MB/s 2023-12-13T23:54:12.782 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-13T23:54:12.839 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 702 Links: 1 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-12-13 23:54:01.976530441 +0000 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-12-13 23:52:06.425584319 +0000 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-12-13 23:52:06.425584319 +0000 2023-12-13T23:54:12.895 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2023-12-13 23:52:06.425584319 +0000 2023-12-13T23:54:12.896 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-13T23:54:12.961 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-12-13T23:54:12.961 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-12-13T23:54:12.961 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000240233 s, 2.1 MB/s 2023-12-13T23:54:12.962 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-13T23:54:13.018 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 720 Links: 1 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2023-12-13T23:54:13.073 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2023-12-13 23:54:01.976530441 +0000 2023-12-13T23:54:13.074 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2023-12-13 23:52:06.939571355 +0000 2023-12-13T23:54:13.074 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2023-12-13 23:52:06.939571355 +0000 2023-12-13T23:54:13.074 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2023-12-13 23:52:06.939571355 +0000 2023-12-13T23:54:13.074 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-13T23:54:13.140 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2023-12-13T23:54:13.140 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2023-12-13T23:54:13.140 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000181152 s, 2.8 MB/s 2023-12-13T23:54:13.141 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-13T23:54:13.197 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:13.197 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-13T23:54:13.197 INFO:tasks.ceph:Generating config... 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] client mount timeout = 600 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] debug client = 20 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] debug ms = 1 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] pid file = 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-12-13T23:54:13.199 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] debug mds = 20 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] debug ms = 1 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] mds debug frag = True 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-12-13T23:54:13.199 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-13T23:54:13.200 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-12-13T23:54:13.200 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-13T23:54:13.200 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-13T23:54:13.200 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-13T23:54:13.200 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-13T23:54:13.201 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-13T23:54:13.202 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-12-13T23:54:13.202 INFO:tasks.ceph:Setting up mon.a... 2023-12-13T23:54:13.202 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-13T23:54:13.252 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.keyring 2023-12-13T23:54:13.254 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-13T23:54:13.298 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-13T23:54:13.365 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.124'), ('mon.b', '172.21.15.203'), ('mon.c', '[v2:172.21.15.203:3301,v1:172.21.15.203:6790]')] 2023-12-13T23:54:13.366 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.124,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-12-13T23:54:13.367 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:13.367 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-13T23:54:13.420 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.124 --add b 172.21.15.203 --addv c '[v2:172.21.15.203:3301,v1:172.21.15.203:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:13.505 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:13.505 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: generated fsid f45f9ea9-91cf-4589-b298-b4d7e1fc4f09 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:setting min_mon_release = quincy 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:epoch 0 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:fsid f45f9ea9-91cf-4589-b298-b4d7e1fc4f09 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:last_changed 2023-12-13T23:54:13.509130+0000 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:created 2023-12-13T23:54:13.509130+0000 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:min_mon_release 17 (quincy) 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:election_strategy: 1 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:0: [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] mon.a 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:1: [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] mon.b 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:2: [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] mon.c 2023-12-13T23:54:13.506 INFO:teuthology.orchestra.run.smithi124.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-13T23:54:13.507 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-13T23:54:13.522 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f45f9ea9-91cf-4589-b298-b4d7e1fc4f09... 2023-12-13T23:54:13.523 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-13T23:54:13.565 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-13T23:54:13.612 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2023-12-13T23:54:13.612 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2023-12-13T23:54:13.612 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-13T23:54:13.612 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2023-12-13T23:54:13.612 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.613 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-13T23:54:13.614 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: mon_allow_pool_size_one = true 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.124,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]" 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = f45f9ea9-91cf-4589-b298-b4d7e1fc4f09 2023-12-13T23:54:13.615 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd mclock profile = high_recovery_ops 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug trim objects = true 2023-12-13T23:54:13.616 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2023-12-13T23:54:13.617 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: bdev async discard = True 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: bdev enable discard = True 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore allocator = bitmap 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore block size = 96636764160 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore fsck on mount = True 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluefs = 1/20 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluestore = 1/20 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2023-12-13T23:54:13.618 INFO:teuthology.orchestra.run.smithi124.stdout: debug rocksdb = 4/10 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd backfillfull_ratio = 0.85 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd full ratio = 0.9 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd nearfull ratio = 0.8 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: osd failsafe full ratio = 0.95 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = bluestore 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-12-13T23:54:13.619 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-13T23:54:13.620 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: # 1m isn't quite enough 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: mon_down_mkfs_grace = 2m 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_filestore_osds = false 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2023-12-13T23:54:13.621 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = "" 2023-12-13T23:54:13.622 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2023-12-13T23:54:13.623 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2023-12-13T23:54:13.623 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2023-12-13T23:54:13.623 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2023-12-13T23:54:13.623 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds balancer = 20 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2023-12-13T23:54:13.624 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2023-12-13T23:54:13.625 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2023-12-13T23:54:13.626 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-13T23:54:13.627 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "172.21.15.124,172.21.15.203,[v2:172.21.15.203:3301,v1:172.21.15.203:6790]" 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = f45f9ea9-91cf-4589-b298-b4d7e1fc4f09 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2023-12-13T23:54:13.628 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2023-12-13T23:54:13.630 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.630 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2023-12-13T23:54:13.630 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2023-12-13T23:54:13.630 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2023-12-13T23:54:13.630 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2023-12-13T23:54:13.631 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: bdev async discard = True 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: bdev enable discard = True 2023-12-13T23:54:13.632 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore allocator = bitmap 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 1/20 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 1/20 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 4/10 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: osd op complaint time = 180 2023-12-13T23:54:13.633 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-12-13T23:54:13.634 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2023-12-13T23:54:13.635 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: mon op complaint time = 120 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2023-12-13T23:54:13.636 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: client mount timeout = 600 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: debug client = 20 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = "" 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: rados mon op timeout = 900 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout: rados osd op timeout = 900 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.a] 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.c] 2023-12-13T23:54:13.637 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.b] 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.d] 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout:[mds] 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: debug mds = 20 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: debug mds balancer = 20 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: mds debug frag = True 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: mds debug scatterstat = True 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: mds op complaint time = 180 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: mds verify scatter = True 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: osd op complaint time = 180 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: rados mon op timeout = 900 2023-12-13T23:54:13.638 INFO:teuthology.orchestra.run.smithi203.stdout: rados osd op timeout = 900 2023-12-13T23:54:13.640 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-13T23:54:13.640 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-13T23:54:13.735 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-13T23:54:13.735 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:13.735 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-13T23:54:13.750 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:13.750 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-13T23:54:13.805 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:54:13.805 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:13.805 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-13T23:54:13.805 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-13T23:54:13.884 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:13.884 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:13.939 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:54:13.940 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:13.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-13T23:54:13.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-13T23:54:13.980 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:13.980 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:14.035 INFO:tasks.ceph:Setting up mon nodes... 2023-12-13T23:54:14.035 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-13T23:54:14.035 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-13T23:54:14.089 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-13T23:54:14.092 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-13T23:54:14.150 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-13T23:54:14.153 INFO:tasks.ceph:Setting up mds nodes... 2023-12-13T23:54:14.153 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-12-13T23:54:14.206 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-12-13T23:54:14.209 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-12-13T23:54:14.235 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-12-13T23:54:14.331 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-12-13T23:54:14.333 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-12-13T23:54:14.359 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-12-13T23:54:14.414 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-12-13T23:54:14.417 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-12-13T23:54:14.443 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-12-13T23:54:14.538 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-12-13T23:54:14.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-12-13T23:54:14.567 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-13T23:54:14.568 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-13T23:54:14.613 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-13T23:54:14.626 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-12-13T23:54:14.708 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-12-13T23:54:14.723 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-13T23:54:14.765 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-13T23:54:14.780 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-12-13T23:54:14.862 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-12-13T23:54:14.878 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-13T23:54:14.878 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-13T23:54:14.878 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-13T23:54:14.905 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:14.905 INFO:tasks.ceph:role: osd.0 2023-12-13T23:54:14.905 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:54:14.905 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:14.979 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:14.980 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:14.980 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:14.980 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:15.470 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-12-13T23:54:15.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-12-13T23:54:15.472 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-13T23:54:15.603 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-13T23:54:15.638 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-13T23:54:15.706 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:15.706 INFO:tasks.ceph:role: osd.1 2023-12-13T23:54:15.706 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:54:15.706 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-13T23:54:15.780 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:15.781 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:16.261 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-12-13T23:54:16.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-12-13T23:54:16.263 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-13T23:54:16.357 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-13T23:54:16.388 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-13T23:54:16.455 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:16.455 INFO:tasks.ceph:role: osd.2 2023-12-13T23:54:16.455 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:54:16.455 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-13T23:54:16.529 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:16.529 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:16.529 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:16.529 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:16.530 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:17.038 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-12-13T23:54:17.040 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-12-13T23:54:17.040 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-13T23:54:17.174 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-13T23:54:17.207 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-13T23:54:17.275 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:17.275 INFO:tasks.ceph:role: osd.3 2023-12-13T23:54:17.275 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:54:17.275 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:17.349 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:17.350 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:17.350 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:17.350 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:17.350 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:17.864 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-12-13T23:54:17.865 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-12-13T23:54:17.866 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-13T23:54:17.971 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-13T23:54:18.005 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:18.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.130+0000 7fb08e307500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-13T23:54:18.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.130+0000 7fb08e307500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-13T23:54:18.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.130+0000 7fb08e307500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-13T23:54:18.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.130+0000 7fb08e307500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-13T23:54:18.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.130+0000 7fb08e307500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-13T23:54:18.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:18.131+0000 7fb08e307500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-13T23:54:20.993 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-13T23:54:21.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.115+0000 7feb44e10500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.116+0000 7feb44e10500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.116+0000 7feb44e10500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.116+0000 7feb44e10500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.116+0000 7feb44e10500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-13T23:54:21.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:21.116+0000 7feb44e10500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-13T23:54:23.960 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-13T23:54:23.988 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.078+0000 7f716bc16500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.078+0000 7f716bc16500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.078+0000 7f716bc16500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.078+0000 7f716bc16500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.078+0000 7f716bc16500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-13T23:54:24.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:24.079+0000 7f716bc16500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-13T23:54:26.919 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-13T23:54:26.948 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:27.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.039+0000 7fe3faaa7500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-13T23:54:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.039+0000 7fe3faaa7500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-13T23:54:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.039+0000 7fe3faaa7500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-13T23:54:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.039+0000 7fe3faaa7500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-13T23:54:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.039+0000 7fe3faaa7500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-13T23:54:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:27.040+0000 7fe3faaa7500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-13T23:54:29.884 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-13T23:54:29.912 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-13T23:54:29.944 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:29.945 INFO:tasks.ceph:role: osd.4 2023-12-13T23:54:29.945 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:54:29.945 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-13T23:54:30.021 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:30.022 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:30.524 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-12-13T23:54:30.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-12-13T23:54:30.526 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-13T23:54:30.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-13T23:54:30.686 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-13T23:54:30.754 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:30.754 INFO:tasks.ceph:role: osd.5 2023-12-13T23:54:30.754 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:54:30.755 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:30.830 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:30.831 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:30.831 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:30.831 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:30.831 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:31.289 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-12-13T23:54:31.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-12-13T23:54:31.291 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-13T23:54:31.375 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-13T23:54:31.446 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-13T23:54:31.513 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:31.513 INFO:tasks.ceph:role: osd.6 2023-12-13T23:54:31.513 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:54:31.513 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:31.589 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:32.086 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-12-13T23:54:32.088 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-12-13T23:54:32.088 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-13T23:54:32.243 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-13T23:54:32.278 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-13T23:54:32.346 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-13T23:54:32.346 INFO:tasks.ceph:role: osd.7 2023-12-13T23:54:32.346 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:54:32.346 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-13T23:54:32.422 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2023-12-13T23:54:32.423 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-13T23:54:32.423 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-13T23:54:32.423 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-13T23:54:32.423 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-13T23:54:32.930 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2023-12-13T23:54:32.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2023-12-13T23:54:32.932 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-13T23:54:33.047 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-13T23:54:33.084 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.190+0000 7f1cd4214500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.191+0000 7f1cd4214500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.191+0000 7f1cd4214500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.191+0000 7f1cd4214500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.191+0000 7f1cd4214500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-13T23:54:33.189 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:33.191+0000 7f1cd4214500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-13T23:54:36.036 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-13T23:54:36.067 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:36.159 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.161+0000 7ff7458ab500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-13T23:54:36.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.162+0000 7ff7458ab500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-13T23:54:36.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.162+0000 7ff7458ab500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-13T23:54:36.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.162+0000 7ff7458ab500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-13T23:54:36.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.162+0000 7ff7458ab500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-13T23:54:36.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:36.162+0000 7ff7458ab500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-13T23:54:39.001 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-13T23:54:39.031 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.124+0000 7f89bc40e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-13T23:54:39.122 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.125+0000 7f89bc40e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-13T23:54:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.125+0000 7f89bc40e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-13T23:54:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.125+0000 7f89bc40e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-13T23:54:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.125+0000 7f89bc40e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-13T23:54:39.123 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:39.125+0000 7f89bc40e500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-13T23:54:41.579 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:41.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:54:41.970 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-13T23:54:41.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:42.086 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.088+0000 7fa2ba8da500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-13T23:54:42.086 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.088+0000 7fa2ba8da500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-13T23:54:42.086 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.088+0000 7fa2ba8da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-13T23:54:42.086 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.088+0000 7fa2ba8da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-13T23:54:42.086 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.088+0000 7fa2ba8da500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-13T23:54:42.087 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:42.089+0000 7fa2ba8da500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-13T23:54:44.937 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-13T23:54:44.965 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-13T23:54:44.965 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:44.966 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-13T23:54:44.996 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:44.996 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-12-13T23:54:45.066 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.066 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-12-13T23:54:45.135 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.135 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-13T23:54:45.204 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.205 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-13T23:54:45.272 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.272 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-13T23:54:45.339 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.340 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-13T23:54:45.407 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.407 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-13T23:54:45.437 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-12-13T23:54:45.505 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.505 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-12-13T23:54:45.573 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.573 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-13T23:54:45.640 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.641 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-13T23:54:45.708 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.708 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-13T23:54:45.777 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.777 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-13T23:54:45.843 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.843 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-13T23:54:45.861 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:45.861 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-12-13T23:54:45.916 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.916 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-13T23:54:45.930 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:45.931 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-12-13T23:54:45.987 INFO:tasks.ceph:Adding keys to all mons... 2023-12-13T23:54:45.987 DEBUG:teuthology.orchestra.run.smithi124:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-13T23:54:45.990 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-13T23:54:46.031 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.y] 2023-12-13T23:54:46.032 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplDCGRBRAATplO/LAqiqpuOu7VOMvxog== 2023-12-13T23:54:46.032 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2023-12-13T23:54:46.032 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplFo6KDBAArJJKNRTjxYdaYwtbHVbu7g== 2023-12-13T23:54:46.032 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHpllD7vExAAj4trq2lGnNXIIi2G5k+/IQ== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.0] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAqRHplcCLWBxAAxYxXKFh6L9WK7uOg5UqRLA== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.1] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAtRHplDW73BhAAQoBtr86nfF318aMqfI/7tA== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.2] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAwRHpl+Bi6BBAAR9+uNWYlp4BCI3gmEbvp3Q== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.3] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAzRHplrYhsAhAADvJtD6bIDoXVG50rEYkYaQ== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.x] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplEAcvCRAAuDLwmskaKHqzdX0/vTGodQ== 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2023-12-13T23:54:46.033 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHpl893rGBAAas73Q+Mux4HJCWgSronQlQ== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHpliKtJIBAAz3MbEgTL5bv/l2zLj32n5w== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.4] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQA5RHpleClsCxAAeVn2LqqB2gIUnF3HAimTnw== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.5] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQA8RHpl552xCRAAgCBrpQroUXTlg+wox8Fx/w== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.6] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQA/RHplFgN+BxAAkUw1AQX9IJpOvNgPFBkQ3g== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.7] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQBCRHplw4NRBRAAbPuwtmaK8/9lrd4crZHHlg== 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:54:46.034 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout:[client.2] 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplGAlqKhAA3u5mk3DyH7KsWsAScLW0Ew== 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout:[client.1] 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplz1vULRAANGEKQl90MW0+VmH0N96EXw== 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout:[client.3] 2023-12-13T23:54:46.035 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHpl7tuiMxAAF4ySYml4t9rxEC9Ji3/wdw== 2023-12-13T23:54:46.053 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2023-12-13T23:54:46.053 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplDCGRBRAATplO/LAqiqpuOu7VOMvxog== 2023-12-13T23:54:46.053 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.a] 2023-12-13T23:54:46.053 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplFo6KDBAArJJKNRTjxYdaYwtbHVbu7g== 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.c] 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHpllD7vExAAj4trq2lGnNXIIi2G5k+/IQ== 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAqRHplcCLWBxAAxYxXKFh6L9WK7uOg5UqRLA== 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2023-12-13T23:54:46.054 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAtRHplDW73BhAAQoBtr86nfF318aMqfI/7tA== 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAwRHpl+Bi6BBAAR9+uNWYlp4BCI3gmEbvp3Q== 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAzRHplrYhsAhAADvJtD6bIDoXVG50rEYkYaQ== 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2023-12-13T23:54:46.055 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplEAcvCRAAuDLwmskaKHqzdX0/vTGodQ== 2023-12-13T23:54:46.056 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.b] 2023-12-13T23:54:46.056 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHpl893rGBAAas73Q+Mux4HJCWgSronQlQ== 2023-12-13T23:54:46.056 INFO:teuthology.orchestra.run.smithi203.stdout:[mds.d] 2023-12-13T23:54:46.056 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHpliKtJIBAAz3MbEgTL5bv/l2zLj32n5w== 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA5RHpleClsCxAAeVn2LqqB2gIUnF3HAimTnw== 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA8RHpl552xCRAAgCBrpQroUXTlg+wox8Fx/w== 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2023-12-13T23:54:46.057 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQA/RHplFgN+BxAAkUw1AQX9IJpOvNgPFBkQ3g== 2023-12-13T23:54:46.058 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2023-12-13T23:54:46.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBCRHplw4NRBRAAbPuwtmaK8/9lrd4crZHHlg== 2023-12-13T23:54:46.058 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2023-12-13T23:54:46.058 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:54:46.058 INFO:teuthology.orchestra.run.smithi203.stdout:[client.2] 2023-12-13T23:54:46.059 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplGAlqKhAA3u5mk3DyH7KsWsAScLW0Ew== 2023-12-13T23:54:46.059 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2023-12-13T23:54:46.060 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHplz1vULRAANGEKQl90MW0+VmH0N96EXw== 2023-12-13T23:54:46.060 INFO:teuthology.orchestra.run.smithi203.stdout:[client.3] 2023-12-13T23:54:46.060 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQAmRHpl7tuiMxAAF4ySYml4t9rxEC9Ji3/wdw== 2023-12-13T23:54:46.062 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-13T23:54:46.079 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-13T23:54:46.154 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.210 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.212 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.267 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.320 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.375 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.377 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.428 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.430 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.482 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-13T23:54:46.484 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-13T23:54:46.535 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.592 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.594 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-12-13T23:54:46.649 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.652 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.704 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.757 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.760 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.811 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-13T23:54:46.865 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:46.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:46.922 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:46.924 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:46.978 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:46.981 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:47.034 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:47.037 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-13T23:54:47.091 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-13T23:54:47.091 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-13T23:54:47.118 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-13T23:54:47.215 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-13T23:54:47.241 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-13T23:54:47.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-13T23:54:47.365 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-13T23:54:47.392 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-13T23:54:47.460 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-13T23:54:47.557 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-13T23:54:47.584 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:47.586 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-13T23:54:47.639 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-13T23:54:47.639 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-13T23:54:47.640 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-12-13T23:54:47.642 INFO:tasks.ceph.mon.a:Started 2023-12-13T23:54:47.642 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-13T23:54:47.642 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-12-13T23:54:47.683 INFO:tasks.ceph.mon.b:Started 2023-12-13T23:54:47.683 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-13T23:54:47.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-12-13T23:54:47.685 INFO:tasks.ceph.mon.c:Started 2023-12-13T23:54:47.685 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-13T23:54:47.685 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-13T23:54:47.685 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-12-13T23:54:47.687 INFO:tasks.ceph.mgr.y:Started 2023-12-13T23:54:47.688 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-13T23:54:47.688 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-12-13T23:54:47.689 INFO:tasks.ceph.mgr.x:Started 2023-12-13T23:54:47.690 INFO:tasks.ceph:Setting crush tunables to default 2023-12-13T23:54:47.690 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd crush tunables default 2023-12-13T23:54:47.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.792+0000 7ff20c690640 1 Processor -- start 2023-12-13T23:54:47.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.792+0000 7ff20c690640 1 -- start start 2023-12-13T23:54:47.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.793+0000 7ff20c690640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:47.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.793+0000 7ff20c690640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:47.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.793+0000 7ff20c690640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20415fc00 con 0x7ff2040cd970 2023-12-13T23:54:47.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.793+0000 7ff20c690640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2040d0730 con 0x7ff2040cb770 2023-12-13T23:54:47.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.793+0000 7ff20c690640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cd110 con 0x7ff20415b130 2023-12-13T23:54:47.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:47.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:47.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:49944/0 (socket says 172.21.15.124:49944) 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:45600/0 (socket says 172.21.15.124:45600) 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 -- 172.21.15.124:0/3339047214 learned_addr learned my addr 172.21.15.124:0/3339047214 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20ac2d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff2040cd970 0x7ff2040cdde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:36232/0 (socket says 172.21.15.124:36232) 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff20415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 -- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 msgr2=0x7ff20415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 -- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 msgr2=0x7ff20415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff20415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-13T23:54:47.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.794+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff209c2b640 1 -- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 msgr2=0x7ff20415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff209c2b640 1 -- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 msgr2=0x7ff20415f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-13T23:54:47.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff20415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-13T23:54:47.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-13T23:54:47.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff20415b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-13T23:54:47.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-13T23:54:47.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:47.995+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-13T23:54:48.002 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.005+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638815890 0 0) 0x7ff20415fc00 con 0x7ff2040cd970 2023-12-13T23:54:48.003 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.005+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ec003520 con 0x7ff2040cd970 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 644319712 0 0) 0x7ff1ec003520 con 0x7ff2040cd970 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff20415b510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff20415b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 msgr2=0x7ff20415f6c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 --2- 172.21.15.124:0/3339047214 >> v2:172.21.15.203:3300/0 conn(0x7ff2040cb770 0x7ff20415f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040cd3b0 con 0x7ff2040cd970 2023-12-13T23:54:48.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7ff1fc001670 con 0x7ff2040cd970 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 >> v1:172.21.15.203:6789/0 conn(0x7ff2040cd970 legacy=0x7ff2040cdde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1ec003970 0x7ff1ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff1ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1ec004d00 0x7ff1ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040d0730 con 0x7ff1ec003970 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cd110 con 0x7ff1ec004d00 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec00bb10 con 0x7ff20415b130 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1ec003970 0x7ff1ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff209c2b640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff1ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20ac2d640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1ec004d00 0x7ff1ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 msgr2=0x7ff1ec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff20415b130 0x7ff1ec0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1ec004d00 msgr2=0x7ff1ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1ec004d00 0x7ff1ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.011+0000 7ff20a42c640 1 -- 172.21.15.124:0/3339047214 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec003520 con 0x7ff1ec003970 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.012+0000 7ff20a42c640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1ec003970 0x7ff1ec003d90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009610 tx=0x7ff1ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b21379b17426d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.012+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4033600 con 0x7ff1ec003970 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.012+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f402f050 con 0x7ff1ec003970 2023-12-13T23:54:48.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.012+0000 7ff20942a640 1 -- 172.21.15.124:0/3339047214 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40339a0 con 0x7ff1ec003970 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 -- 172.21.15.124:0/3339047214 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1ec003970 msgr2=0x7ff1ec003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 --2- 172.21.15.124:0/3339047214 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1ec003970 0x7ff1ec003d90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009610 tx=0x7ff1ec0057f0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20ac2d640 1 -- 172.21.15.124:0/3339047214 reap_dead start 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 -- 172.21.15.124:0/3339047214 shutdown_connections 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 -- 172.21.15.124:0/3339047214 >> 172.21.15.124:0/3339047214 conn(0x7ff2040b2c10 msgr2=0x7ff2040b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 -- 172.21.15.124:0/3339047214 shutdown_connections 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.013+0000 7ff20c690640 1 -- 172.21.15.124:0/3339047214 wait complete. 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.014+0000 7ff20c690640 1 Processor -- start 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.014+0000 7ff20c690640 1 -- start start 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1ec003970 0x7ff2040c7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff20415d270 0x7ff20415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20415dc30 con 0x7ff2040c4b70 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040b4620 con 0x7ff20415d270 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20c690640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040c7e40 con 0x7ff1ec003970 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:34388/0 (socket says 172.21.15.124:34388) 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 -- 172.21.15.124:0/913252072 learned_addr learned my addr 172.21.15.124:0/913252072 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20a42c640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1ec003970 0x7ff2040c7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff20ac2d640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff20415d270 0x7ff20415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 -- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1ec003970 msgr2=0x7ff2040c7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1ec003970 0x7ff2040c7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 -- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff20415d270 msgr2=0x7ff20415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff20415d270 0x7ff20415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 -- 172.21.15.124:0/913252072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040b3cf0 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.015+0000 7ff209c2b640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8013750 tx=0x7ff1f8013c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8004440 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f800c070 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040b3fd0 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8011a00 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20414a920 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7ff1f8011b60 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.016+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff1f8003cc0 con 0x7ff2040c4b70 2023-12-13T23:54:48.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.017+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff204067fc0 con 0x7ff2040c4b70 2023-12-13T23:54:48.018 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.019+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7ff1f8024060 con 0x7ff2040c4b70 2023-12-13T23:54:48.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.148+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff204067420 con 0x7ff2040c4b70 2023-12-13T23:54:48.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.149+0000 7ff202ffd640 1 -- 172.21.15.124:0/913252072 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff1f801a070 con 0x7ff2040c4b70 2023-12-13T23:54:48.147 INFO:teuthology.orchestra.run.smithi124.stderr:adjusted tunables profile to default 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 msgr2=0x7ff20414a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1f8013750 tx=0x7ff1f8013c10 comp rx=0 tx=0).stop 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 shutdown_connections 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff20415d270 0x7ff20415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2040c4b70 0x7ff20414a3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 --2- 172.21.15.124:0/913252072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1ec003970 0x7ff2040c7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 >> 172.21.15.124:0/913252072 conn(0x7ff2040b2c10 msgr2=0x7ff2040cbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:48.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 shutdown_connections 2023-12-13T23:54:48.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:48.151+0000 7ff20c690640 1 -- 172.21.15.124:0/913252072 wait complete. 2023-12-13T23:54:48.160 INFO:tasks.ceph:check_enable_crimson: False 2023-12-13T23:54:48.161 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-13T23:54:48.161 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:48.161 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-13T23:54:48.188 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:48.188 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-13T23:54:48.255 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:48.255 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-13T23:54:48.322 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:54:48.323 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-13T23:54:48.389 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:48.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-13T23:54:48.418 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:48.418 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-13T23:54:48.449 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:48.451+0000 7f2fbb616040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-13T23:54:48.475 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:48.476 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-13T23:54:48.503 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:48.505+0000 7f2fbb616040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-13T23:54:48.530 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:54:48.530 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-13T23:54:48.532 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:48.534+0000 7efe50a37040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-13T23:54:48.583 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:48.585+0000 7efe50a37040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-13T23:54:48.596 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 27b9dad0-42a9-4d72-8917-1fef77b307ab 0 2023-12-13T23:54:48.737 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.739+0000 7efd11501640 1 Processor -- start 2023-12-13T23:54:48.737 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.739+0000 7efd11501640 1 -- start start 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd11501640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efd0c0b5370 0x7efd0c0cda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd11501640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c0d1bb0 con 0x7efd0c0d2040 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd11501640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd0c15d6b0 con 0x7efd0c15d2d0 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd11501640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd0c0ce700 con 0x7efd0c0b5370 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efd0c0b5370 0x7efd0c0cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7efd0c0b5370 0x7efd0c0cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48898/0 (socket says 172.21.15.203:48898) 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 -- 172.21.15.203:0/4104895622 learned_addr learned my addr 172.21.15.203:0/4104895622 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:48.740 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 -- 172.21.15.203:0/4104895622 >> v1:172.21.15.203:6789/0 conn(0x7efd0c15d2d0 legacy=0x7efd0c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 -- 172.21.15.203:0/4104895622 >> v1:172.21.15.124:6789/0 conn(0x7efd0c0d2040 legacy=0x7efd0c0b6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.740+0000 7efd0a7fc640 1 -- 172.21.15.203:0/4104895622 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c0d1280 con 0x7efd0c0b5370 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd0a7fc640 1 --2- 172.21.15.203:0/4104895622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cda40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009870 tx=0x7efcfc02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e6b1ca99aa47a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd09ffb640 1 -- 172.21.15.203:0/4104895622 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc030860 con 0x7efd0c0b5370 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd09ffb640 1 -- 172.21.15.203:0/4104895622 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc0309c0 con 0x7efd0c0b5370 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd09ffb640 1 -- 172.21.15.203:0/4104895622 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc030d50 con 0x7efd0c0b5370 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 -- 172.21.15.203:0/4104895622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0b5370 msgr2=0x7efd0c0cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 --2- 172.21.15.203:0/4104895622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cda40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009870 tx=0x7efcfc02b9a0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 -- 172.21.15.203:0/4104895622 shutdown_connections 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 --2- 172.21.15.203:0/4104895622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cda40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 -- 172.21.15.203:0/4104895622 >> 172.21.15.203:0/4104895622 conn(0x7efd0c072050 msgr2=0x7efd0c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:48.741 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 -- 172.21.15.203:0/4104895622 shutdown_connections 2023-12-13T23:54:48.742 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.741+0000 7efd11501640 1 -- 172.21.15.203:0/4104895622 wait complete. 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 Processor -- start 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 -- start start 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cb300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd0c15d2d0 0x7efd0c1587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0cb840 con 0x7efd0c0b5370 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0cba70 con 0x7efd0c0d2040 2023-12-13T23:54:48.744 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.743+0000 7efd11501640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd0c0cbd50 con 0x7efd0c15d2d0 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48904/0 (socket says 172.21.15.203:48904) 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 -- 172.21.15.203:0/3804214378 learned_addr learned my addr 172.21.15.203:0/3804214378 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0b7fe640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd0c15d2d0 0x7efd0c1587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0affd640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 -- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd0c15d2d0 msgr2=0x7efd0c1587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd0c15d2d0 0x7efd0c1587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 -- 172.21.15.203:0/3804214378 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efd0c0b5370 msgr2=0x7efd0c0cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cb300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 -- 172.21.15.203:0/3804214378 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc0094d0 con 0x7efd0c0d2040 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd0a7fc640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009870 tx=0x7efcfc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.745 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc030810 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcfc043400 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd0c0cc030 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0437b0 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd0c0cc5f0 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7efcfc043c90 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.744+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7efcfc03e070 con 0x7efd0c0d2040 2023-12-13T23:54:48.746 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.745+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd0c069400 con 0x7efd0c0d2040 2023-12-13T23:54:48.747 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.747+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7efcfc034070 con 0x7efd0c0d2040 2023-12-13T23:54:48.817 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.819+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "27b9dad0-42a9-4d72-8917-1fef77b307ab", "id": 0} v 0) v1 -- 0x7efd0c1606c0 con 0x7efd0c0d2040 2023-12-13T23:54:48.820 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.822+0000 7efcf37fe640 1 -- 172.21.15.203:0/3804214378 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "27b9dad0-42a9-4d72-8917-1fef77b307ab", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7efcfc04d590 con 0x7efd0c0d2040 2023-12-13T23:54:48.820 INFO:teuthology.orchestra.run.smithi203.stdout:0 2023-12-13T23:54:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 msgr2=0x7efd0c15b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcfc009870 tx=0x7efcfc004730 comp rx=0 tx=0).stop 2023-12-13T23:54:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 shutdown_connections 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7efd0c0d2040 0x7efd0c15b580 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7efd0c15d2d0 0x7efd0c1587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 --2- 172.21.15.203:0/3804214378 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efd0c0b5370 0x7efd0c0cb300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 >> 172.21.15.203:0/3804214378 conn(0x7efd0c072050 msgr2=0x7efd0c0b7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 shutdown_connections 2023-12-13T23:54:48.822 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.823+0000 7efd11501640 1 -- 172.21.15.203:0/3804214378 wait complete. 2023-12-13T23:54:48.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new bcdb3444-af55-4d8e-9e7b-d8f23ea24f51 1 2023-12-13T23:54:48.932 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.934+0000 7f23bd6cf640 1 Processor -- start 2023-12-13T23:54:48.932 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.934+0000 7f23bd6cf640 1 -- start start 2023-12-13T23:54:48.934 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23bd6cf640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23b81432a0 0x7f23b815d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.934 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23bd6cf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f23b806abe0 0x7f23b81409d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23bd6cf640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b806e960 con 0x7f23b8143680 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23bd6cf640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b815dec0 con 0x7f23b81432a0 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23bd6cf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b815e1a0 con 0x7f23b806abe0 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f23b806abe0 0x7f23b81409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f23b806abe0 0x7f23b81409d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48918/0 (socket says 172.21.15.203:48918) 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 -- 172.21.15.203:0/2134870827 learned_addr learned my addr 172.21.15.203:0/2134870827 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b6ffd640 1 --2- 172.21.15.203:0/2134870827 >> v2:172.21.15.124:3300/0 conn(0x7f23b81432a0 0x7f23b815d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 -- 172.21.15.203:0/2134870827 >> v2:172.21.15.124:3300/0 conn(0x7f23b81432a0 msgr2=0x7f23b815d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:48.935 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 --2- 172.21.15.203:0/2134870827 >> v2:172.21.15.124:3300/0 conn(0x7f23b81432a0 0x7f23b815d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 -- 172.21.15.203:0/2134870827 >> v1:172.21.15.124:6789/0 conn(0x7f23b8143680 legacy=0x7f23b806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b77fe640 1 -- 172.21.15.203:0/2134870827 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b815e480 con 0x7f23b806abe0 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.935+0000 7f23b5ffb640 1 -- 172.21.15.203:0/2134870827 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4068558738 0 0) 0x7f23b806e960 con 0x7f23b8143680 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23b77fe640 1 --2- 172.21.15.203:0/2134870827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b81409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23b80ace00 tx=0x7f23b002bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd7b1a4f4411a99 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23b5ffb640 1 -- 172.21.15.203:0/2134870827 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b002cb20 con 0x7f23b806abe0 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23b5ffb640 1 -- 172.21.15.203:0/2134870827 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b002cc80 con 0x7f23b806abe0 2023-12-13T23:54:48.936 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23b5ffb640 1 -- 172.21.15.203:0/2134870827 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b00343f0 con 0x7f23b806abe0 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 -- 172.21.15.203:0/2134870827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 msgr2=0x7f23b81409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/2134870827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b81409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23b80ace00 tx=0x7f23b002bf80 comp rx=0 tx=0).stop 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 -- 172.21.15.203:0/2134870827 shutdown_connections 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/2134870827 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b81409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/2134870827 >> v2:172.21.15.124:3300/0 conn(0x7f23b81432a0 0x7f23b815d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 -- 172.21.15.203:0/2134870827 >> 172.21.15.203:0/2134870827 conn(0x7f23b80b1de0 msgr2=0x7f23b80b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 -- 172.21.15.203:0/2134870827 shutdown_connections 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.937+0000 7f23bd6cf640 1 -- 172.21.15.203:0/2134870827 wait complete. 2023-12-13T23:54:48.940 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 Processor -- start 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 -- start start 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23b81432a0 0x7f23b81f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 0x7f23b8160980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b81f74a0 con 0x7f23b81432a0 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8131570 con 0x7f23b806abe0 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.939+0000 7f23bd6cf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8130cd0 con 0x7f23b8143680 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.941 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48934/0 (socket says 172.21.15.203:48934) 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 -- 172.21.15.203:0/1077258396 learned_addr learned my addr 172.21.15.203:0/1077258396 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b77fe640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 0x7f23b8160980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b67fc640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23b81432a0 0x7f23b81f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 -- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 msgr2=0x7f23b8160980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 0x7f23b8160980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 -- 172.21.15.203:0/1077258396 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23b81432a0 msgr2=0x7f23b81f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23b81432a0 0x7f23b81f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 -- 172.21.15.203:0/1077258396 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b0009b30 con 0x7f23b806abe0 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b77fe640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 0x7f23b8160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23b6ffd640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00e9b0 tx=0x7f23ac00ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac009670 con 0x7f23b806abe0 2023-12-13T23:54:48.942 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00c9a0 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b8130fb0 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac00cd50 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b81f3070 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f23ac005990 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.940+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f23ac013820 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.941+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b8067420 con 0x7f23b806abe0 2023-12-13T23:54:48.943 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:48.945+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f23ac021060 con 0x7f23b806abe0 2023-12-13T23:54:49.015 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.017+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "bcdb3444-af55-4d8e-9e7b-d8f23ea24f51", "id": 1} v 0) v1 -- 0x7f23b80b3630 con 0x7f23b806abe0 2023-12-13T23:54:49.021 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.023+0000 7f239f7fe640 1 -- 172.21.15.203:0/1077258396 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bcdb3444-af55-4d8e-9e7b-d8f23ea24f51", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f23ac017070 con 0x7f23b806abe0 2023-12-13T23:54:49.021 INFO:teuthology.orchestra.run.smithi203.stdout:1 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 msgr2=0x7f23b8130610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00e9b0 tx=0x7f23ac00ee70 comp rx=0 tx=0).stop 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 shutdown_connections 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23b8143680 0x7f23b8160980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23b81432a0 0x7f23b81f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 --2- 172.21.15.203:0/1077258396 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23b806abe0 0x7f23b8130610 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 >> 172.21.15.203:0/1077258396 conn(0x7f23b80b1de0 msgr2=0x7f23b813f950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 shutdown_connections 2023-12-13T23:54:49.022 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.024+0000 7f23bd6cf640 1 -- 172.21.15.203:0/1077258396 wait complete. 2023-12-13T23:54:49.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 72e1a62d-472d-4db0-98c0-ff0c9e4e6d86 2 2023-12-13T23:54:49.078 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.080+0000 7f2fbb616040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-13T23:54:49.118 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.121+0000 7efe50a37040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-13T23:54:49.133 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.135+0000 7f2fbb616040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-13T23:54:49.133 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.135+0000 7f85cd89b640 1 Processor -- start 2023-12-13T23:54:49.133 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.135+0000 7f85cd89b640 1 -- start start 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85cd89b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f85c815b080 0x7f85c80cb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85cd89b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f85c80cce90 0x7f85c815f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85cd89b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c815efd0 con 0x7f85c815b460 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85cd89b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c80cf4c0 con 0x7f85c80cce90 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85cd89b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c80cdbc0 con 0x7f85c815b080 2023-12-13T23:54:49.135 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f85c815b080 0x7f85c80cb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f85c815b080 0x7f85c80cb890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48938/0 (socket says 172.21.15.203:48938) 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 -- 172.21.15.203:0/2715209919 learned_addr learned my addr 172.21.15.203:0/2715209919 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c77fe640 1 --2- 172.21.15.203:0/2715209919 >> v2:172.21.15.124:3300/0 conn(0x7f85c80cce90 0x7f85c815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 -- 172.21.15.203:0/2715209919 >> v2:172.21.15.124:3300/0 conn(0x7f85c80cce90 msgr2=0x7f85c815f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 --2- 172.21.15.203:0/2715209919 >> v2:172.21.15.124:3300/0 conn(0x7f85c80cce90 0x7f85c815f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 -- 172.21.15.203:0/2715209919 >> v1:172.21.15.124:6789/0 conn(0x7f85c815b460 legacy=0x7f85c80b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.136+0000 7f85c6ffd640 1 -- 172.21.15.203:0/2715209919 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c80cd320 con 0x7f85c815b080 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.137+0000 7f85c6ffd640 1 --2- 172.21.15.203:0/2715209919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b080 0x7f85c80cb890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85c80720f0 tx=0x7f85b802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=504dd3d209e509ac server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.137+0000 7f85c5ffb640 1 -- 172.21.15.203:0/2715209919 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8030860 con 0x7f85c815b080 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.137+0000 7f85c5ffb640 1 -- 172.21.15.203:0/2715209919 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b80309c0 con 0x7f85c815b080 2023-12-13T23:54:49.136 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.137+0000 7f85c5ffb640 1 -- 172.21.15.203:0/2715209919 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8030d50 con 0x7f85c815b080 2023-12-13T23:54:49.137 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 -- 172.21.15.203:0/2715209919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b080 msgr2=0x7f85c80cb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.137 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 --2- 172.21.15.203:0/2715209919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b080 0x7f85c80cb890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f85c80720f0 tx=0x7f85b802b9a0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.137 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 -- 172.21.15.203:0/2715209919 shutdown_connections 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 --2- 172.21.15.203:0/2715209919 >> v2:172.21.15.124:3300/0 conn(0x7f85c80cce90 0x7f85c815f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 --2- 172.21.15.203:0/2715209919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b080 0x7f85c80cb890 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 -- 172.21.15.203:0/2715209919 >> 172.21.15.203:0/2715209919 conn(0x7f85c8070ec0 msgr2=0x7f85c80712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 -- 172.21.15.203:0/2715209919 shutdown_connections 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.138+0000 7f85cd89b640 1 -- 172.21.15.203:0/2715209919 wait complete. 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.139+0000 7f85cd89b640 1 Processor -- start 2023-12-13T23:54:49.138 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.139+0000 7f85cd89b640 1 -- start start 2023-12-13T23:54:49.141 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.141 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 0x7f85c8140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.141 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 0x7f85c8150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c8153d10 con 0x7f85c8152890 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c8141520 con 0x7f85c815b460 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85cd89b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c8153100 con 0x7f85c815b080 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c67fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 0x7f85c8140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c6ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:33990/0 (socket says 172.21.15.203:33990) 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c6ffd640 1 -- 172.21.15.203:0/824435758 learned_addr learned my addr 172.21.15.203:0/824435758 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c77fe640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 0x7f85c8150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.140+0000 7f85c6ffd640 1 -- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 msgr2=0x7f85c8140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c6ffd640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 0x7f85c8140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c6ffd640 1 -- 172.21.15.203:0/824435758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 msgr2=0x7f85c8150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.142 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c6ffd640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 0x7f85c8150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c6ffd640 1 -- 172.21.15.203:0/824435758 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80094d0 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c67fc640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 0x7f85c8140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c77fe640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 0x7f85c8150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85c6ffd640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85b802b9e0 tx=0x7f85b8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8039c90 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8039e10 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c81533e0 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8043640 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c81539a0 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f85b8043b20 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.141+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f85b803e070 con 0x7f85c815b080 2023-12-13T23:54:49.143 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.142+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c8067420 con 0x7f85c815b080 2023-12-13T23:54:49.144 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.144+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f85b8034070 con 0x7f85c815b080 2023-12-13T23:54:49.170 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.173+0000 7efe50a37040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-13T23:54:49.180 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.182+0000 7f2fbb616040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-13T23:54:49.214 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.216+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "72e1a62d-472d-4db0-98c0-ff0c9e4e6d86", "id": 2} v 0) v1 -- 0x7f85c806f6a0 con 0x7f85c815b080 2023-12-13T23:54:49.215 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.217+0000 7efe50a37040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-13T23:54:49.219 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.221+0000 7f85affff640 1 -- 172.21.15.203:0/824435758 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72e1a62d-472d-4db0-98c0-ff0c9e4e6d86", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f85b804d460 con 0x7f85c815b080 2023-12-13T23:54:49.219 INFO:teuthology.orchestra.run.smithi203.stdout:2 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 msgr2=0x7f85c815e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f85b802b9e0 tx=0x7f85b8004730 comp rx=0 tx=0).stop 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 shutdown_connections 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85c815b460 0x7f85c8140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85c8152890 0x7f85c8150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 --2- 172.21.15.203:0/824435758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85c815b080 0x7f85c815e250 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 >> 172.21.15.203:0/824435758 conn(0x7f85c8070ec0 msgr2=0x7f85c81500a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 shutdown_connections 2023-12-13T23:54:49.221 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.222+0000 7f85cd89b640 1 -- 172.21.15.203:0/824435758 wait complete. 2023-12-13T23:54:49.230 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 12ee35f6-b6ed-4265-9e69-d9de43ea99fc 3 2023-12-13T23:54:49.237 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.239+0000 7f2fbb616040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-13T23:54:49.268 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.271+0000 7efe50a37040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-13T23:54:49.281 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.283+0000 7f2fbb616040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-13T23:54:49.311 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.313+0000 7efe50a37040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-13T23:54:49.330 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.332+0000 7f3276ebb640 1 Processor -- start 2023-12-13T23:54:49.331 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.332+0000 7f3276ebb640 1 -- start start 2023-12-13T23:54:49.331 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f3276ebb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3270070540 0x7f327015ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f3276ebb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327015b640 con 0x7f3270155960 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f3276ebb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327015f980 con 0x7f327015c370 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f3276ebb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f327015fc60 con 0x7f3270070540 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f326ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3270070540 0x7f327015ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f326ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3270070540 0x7f327015ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48946/0 (socket says 172.21.15.203:48946) 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.333+0000 7f326ffff640 1 -- 172.21.15.203:0/1752231709 learned_addr learned my addr 172.21.15.203:0/1752231709 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.334+0000 7f326ffff640 1 -- 172.21.15.203:0/1752231709 >> v1:172.21.15.203:6789/0 conn(0x7f327015c370 legacy=0x7f327015f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.332 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.334+0000 7f326ffff640 1 -- 172.21.15.203:0/1752231709 >> v1:172.21.15.124:6789/0 conn(0x7f3270155960 legacy=0x7f327006fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.333 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.334+0000 7f326f7fe640 1 -- 172.21.15.203:0/1752231709 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188483819 0 0) 0x7f327015b640 con 0x7f3270155960 2023-12-13T23:54:49.333 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.334+0000 7f326ffff640 1 -- 172.21.15.203:0/1752231709 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327015b890 con 0x7f3270070540 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.334+0000 7f326ffff640 1 --2- 172.21.15.203:0/1752231709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270070540 0x7f327015ec90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f325c009870 tx=0x7f325c02ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3d9433e0a7398b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f326f7fe640 1 -- 172.21.15.203:0/1752231709 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c02f840 con 0x7f3270070540 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f326f7fe640 1 -- 172.21.15.203:0/1752231709 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f325c02f9a0 con 0x7f3270070540 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f326f7fe640 1 -- 172.21.15.203:0/1752231709 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325c02fd30 con 0x7f3270070540 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 -- 172.21.15.203:0/1752231709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270070540 msgr2=0x7f327015ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 --2- 172.21.15.203:0/1752231709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270070540 0x7f327015ec90 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f325c009870 tx=0x7f325c02ba20 comp rx=0 tx=0).stop 2023-12-13T23:54:49.334 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 -- 172.21.15.203:0/1752231709 shutdown_connections 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 --2- 172.21.15.203:0/1752231709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270070540 0x7f327015ec90 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 -- 172.21.15.203:0/1752231709 >> 172.21.15.203:0/1752231709 conn(0x7f327006e7d0 msgr2=0x7f327006ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 -- 172.21.15.203:0/1752231709 shutdown_connections 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.335+0000 7f3276ebb640 1 -- 172.21.15.203:0/1752231709 wait complete. 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.336+0000 7f3276ebb640 1 Processor -- start 2023-12-13T23:54:49.335 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.336+0000 7f3276ebb640 1 -- start start 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 0x7f32700b19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f327015c370 0x7f32700b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32701f0db0 con 0x7f327015c370 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32701f1090 con 0x7f3270155960 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3276ebb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32701f1370 con 0x7f3270070540 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3274c57640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3274c57640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:34004/0 (socket says 172.21.15.203:34004) 2023-12-13T23:54:49.337 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f3274c57640 1 -- 172.21.15.203:0/2612262496 learned_addr learned my addr 172.21.15.203:0/2612262496 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.337+0000 7f326ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 0x7f32700b19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3275458640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f327015c370 0x7f32700b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 -- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 msgr2=0x7f32700b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 0x7f32700b19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 -- 172.21.15.203:0/2612262496 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f327015c370 msgr2=0x7f32700b1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f327015c370 0x7f32700b1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 -- 172.21.15.203:0/2612262496 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c0094d0 con 0x7f3270070540 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326ffff640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 0x7f32700b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3274c57640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f326400c4b0 tx=0x7f326400c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264005480 con 0x7f3270070540 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32640035f0 con 0x7f3270070540 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32701f1650 con 0x7f3270070540 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32640039e0 con 0x7f3270070540 2023-12-13T23:54:49.338 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32700c9900 con 0x7f3270070540 2023-12-13T23:54:49.339 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f326401c440 con 0x7f3270070540 2023-12-13T23:54:49.339 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.338+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f3264012b60 con 0x7f3270070540 2023-12-13T23:54:49.339 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.339+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32701f1b70 con 0x7f3270070540 2023-12-13T23:54:49.341 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.343+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f3264017070 con 0x7f3270070540 2023-12-13T23:54:49.389 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.391+0000 7f2fbb616040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-13T23:54:49.412 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.414+0000 7efe50a37040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-13T23:54:49.413 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.415+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "12ee35f6-b6ed-4265-9e69-d9de43ea99fc", "id": 3} v 0) v1 -- 0x7f3270067420 con 0x7f3270070540 2023-12-13T23:54:49.418 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.420+0000 7f326d7fa640 1 -- 172.21.15.203:0/2612262496 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "12ee35f6-b6ed-4265-9e69-d9de43ea99fc", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3264012df0 con 0x7f3270070540 2023-12-13T23:54:49.418 INFO:teuthology.orchestra.run.smithi203.stdout:3 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 msgr2=0x7f3270154d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f326400c4b0 tx=0x7f326400c970 comp rx=0 tx=0).stop 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 shutdown_connections 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3270155960 0x7f32700b19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f327015c370 0x7f32700b1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 --2- 172.21.15.203:0/2612262496 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3270070540 0x7f3270154d30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 >> 172.21.15.203:0/2612262496 conn(0x7f327006e7d0 msgr2=0x7f327006ebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.419 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 shutdown_connections 2023-12-13T23:54:49.420 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.421+0000 7f3276ebb640 1 -- 172.21.15.203:0/2612262496 wait complete. 2023-12-13T23:54:49.429 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b 4 2023-12-13T23:54:49.462 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.463+0000 7f2fbb616040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-13T23:54:49.481 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.483+0000 7efe50a37040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-13T23:54:49.510 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.512+0000 7f2fbb616040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-13T23:54:49.526 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.528+0000 7efe50a37040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-13T23:54:49.562 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.564+0000 7fdfee87c640 1 Processor -- start 2023-12-13T23:54:49.563 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.564+0000 7fdfee87c640 1 -- start start 2023-12-13T23:54:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfee87c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfee87c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.564 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfee87c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe806a570 con 0x7fdfe8160f00 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfee87c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe806cc50 con 0x7fdfe8161810 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfee87c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe815e1d0 con 0x7fdfe80b2e30 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfe7fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfece19640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfece19640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:34018/0 (socket says 172.21.15.203:34018) 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfece19640 1 -- 172.21.15.203:0/3926868712 learned_addr learned my addr 172.21.15.203:0/3926868712 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.565+0000 7fdfe7fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48962/0 (socket says 172.21.15.203:48962) 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfece19640 1 -- 172.21.15.203:0/3926868712 >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 msgr2=0x7fdfe815d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:49.565 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfece19640 1 --2- 172.21.15.203:0/3926868712 >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfece19640 1 -- 172.21.15.203:0/3926868712 >> v1:172.21.15.124:6789/0 conn(0x7fdfe8160f00 legacy=0x7fdfe80b2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfece19640 1 -- 172.21.15.203:0/3926868712 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe815e4b0 con 0x7fdfe80b2e30 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfe7fff640 1 --2- 172.21.15.203:0/3926868712 >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.566+0000 7fdfece19640 1 --2- 172.21.15.203:0/3926868712 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009e50 tx=0x7fdfe002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f5d53aeb2cdd3f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfe6ffd640 1 -- 172.21.15.203:0/3926868712 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe002cb00 con 0x7fdfe80b2e30 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfe6ffd640 1 -- 172.21.15.203:0/3926868712 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfe002cc60 con 0x7fdfe80b2e30 2023-12-13T23:54:49.566 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfe6ffd640 1 -- 172.21.15.203:0/3926868712 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfe00343f0 con 0x7fdfe80b2e30 2023-12-13T23:54:49.566 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.567+0000 7f2fbb616040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-13T23:54:49.569 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 -- 172.21.15.203:0/3926868712 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 msgr2=0x7fdfe806de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.569 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 --2- 172.21.15.203:0/3926868712 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfe0009e50 tx=0x7fdfe002bf60 comp rx=0 tx=0).stop 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 -- 172.21.15.203:0/3926868712 shutdown_connections 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 --2- 172.21.15.203:0/3926868712 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe806de80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 --2- 172.21.15.203:0/3926868712 >> v2:172.21.15.203:3300/0 conn(0x7fdfe8161810 0x7fdfe815d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 -- 172.21.15.203:0/3926868712 >> 172.21.15.203:0/3926868712 conn(0x7fdfe80b1a50 msgr2=0x7fdfe80cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 -- 172.21.15.203:0/3926868712 shutdown_connections 2023-12-13T23:54:49.570 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.567+0000 7fdfee87c640 1 -- 172.21.15.203:0/3926868712 wait complete. 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.568+0000 7fdfee87c640 1 Processor -- start 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 -- start start 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe8160f00 0x7fdfe813ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe8158550 con 0x7fdfe8160f00 2023-12-13T23:54:49.571 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe81f3d00 con 0x7fdfe8161810 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfee87c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe80cc370 con 0x7fdfe80b2e30 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfece19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfece19640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48978/0 (socket says 172.21.15.203:48978) 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.569+0000 7fdfe7fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:34028/0 (socket says 172.21.15.203:34028) 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfece19640 1 -- 172.21.15.203:0/603638931 learned_addr learned my addr 172.21.15.203:0/603638931 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe77fe640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe8160f00 0x7fdfe813ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.572 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 -- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 msgr2=0x7fdfe81f37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 -- 172.21.15.203:0/603638931 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe8160f00 msgr2=0x7fdfe813ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe8160f00 0x7fdfe813ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 -- 172.21.15.203:0/603638931 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfdc009400 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfece19640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe7fff640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00e870 tx=0x7fdfdc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc00c940 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfdc00caa0 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc00ce30 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe0009b30 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe81f6bd0 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7fdfdc005990 con 0x7fdfe80b2e30 2023-12-13T23:54:49.573 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.570+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fdfdc013980 con 0x7fdfe80b2e30 2023-12-13T23:54:49.574 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.571+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe8067420 con 0x7fdfe80b2e30 2023-12-13T23:54:49.574 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.573+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7fdfdc021060 con 0x7fdfe80b2e30 2023-12-13T23:54:49.579 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.581+0000 7efe50a37040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-13T23:54:49.614 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:49.616+0000 7f2fbb616040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-13T23:54:49.625 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:49.627+0000 7efe50a37040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-13T23:54:49.642 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.645+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b", "id": 4} v 0) v1 -- 0x7fdfe80d0170 con 0x7fdfe80b2e30 2023-12-13T23:54:49.647 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.650+0000 7fdfe4ff9640 1 -- 172.21.15.203:0/603638931 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdfdc017070 con 0x7fdfe80b2e30 2023-12-13T23:54:49.648 INFO:teuthology.orchestra.run.smithi203.stdout:4 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 msgr2=0x7fdfe81f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc00e870 tx=0x7fdfdc00ed30 comp rx=0 tx=0).stop 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 shutdown_connections 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfe8161810 0x7fdfe81f37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe8160f00 0x7fdfe813ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 --2- 172.21.15.203:0/603638931 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfe80b2e30 0x7fdfe81f6310 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 >> 172.21.15.203:0/603638931 conn(0x7fdfe80b1a50 msgr2=0x7fdfe80af390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 shutdown_connections 2023-12-13T23:54:49.649 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.651+0000 7fdfee87c640 1 -- 172.21.15.203:0/603638931 wait complete. 2023-12-13T23:54:49.659 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new e7ce641b-9ed7-4361-a39b-5551b768c71e 5 2023-12-13T23:54:49.759 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.761+0000 7eff4fa48640 1 Processor -- start 2023-12-13T23:54:49.759 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.761+0000 7eff4fa48640 1 -- start start 2023-12-13T23:54:49.760 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4fa48640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7eff48143820 0x7eff480ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.760 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4fa48640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff480b4180 con 0x7eff48071970 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4fa48640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff480cd430 con 0x7eff480714c0 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4fa48640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4815e8e0 con 0x7eff48143820 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4cfe3640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7eff480714c0 0x7eff4815dac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:33514/0 (socket says 172.21.15.203:33514) 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4cfe3640 1 -- 172.21.15.203:0/2414217922 learned_addr learned my addr 172.21.15.203:0/2414217922 (peer_addr_for_me v1:172.21.15.203:0/0) 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4d7e4640 1 --2- 172.21.15.203:0/2414217922 >> v2:172.21.15.124:3300/0 conn(0x7eff48143820 0x7eff480ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738469723 0 0) 0x7eff480cd430 con 0x7eff480714c0 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34003520 con 0x7eff480714c0 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706957725 0 0) 0x7eff480b4180 con 0x7eff48071970 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff480cd430 con 0x7eff48071970 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4d7e4640 1 -- 172.21.15.203:0/2414217922 >> v1:172.21.15.203:6789/0 conn(0x7eff480714c0 legacy=0x7eff4815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4d7e4640 1 -- 172.21.15.203:0/2414217922 >> v1:172.21.15.124:6789/0 conn(0x7eff48071970 legacy=0x7eff4815e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.761 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.762+0000 7eff4d7e4640 1 -- 172.21.15.203:0/2414217922 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff480cf8b0 con 0x7eff48143820 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.763+0000 7eff4d7e4640 1 --2- 172.21.15.203:0/2414217922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff48143820 0x7eff480ccac0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff440097a0 tx=0x7eff4402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f82daa479b599fdf server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.763+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44030610 con 0x7eff48143820 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.763+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff44030770 con 0x7eff48143820 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.763+0000 7eff3ffff640 1 -- 172.21.15.203:0/2414217922 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff44030b00 con 0x7eff48143820 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 -- 172.21.15.203:0/2414217922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff48143820 msgr2=0x7eff480ccac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.766 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 --2- 172.21.15.203:0/2414217922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff48143820 0x7eff480ccac0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff440097a0 tx=0x7eff4402b960 comp rx=0 tx=0).stop 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 -- 172.21.15.203:0/2414217922 shutdown_connections 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 --2- 172.21.15.203:0/2414217922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff48143820 0x7eff480ccac0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 -- 172.21.15.203:0/2414217922 >> 172.21.15.203:0/2414217922 conn(0x7eff480b1de0 msgr2=0x7eff480b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 -- 172.21.15.203:0/2414217922 shutdown_connections 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.764+0000 7eff4fa48640 1 -- 172.21.15.203:0/2414217922 wait complete. 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.765+0000 7eff4fa48640 1 Processor -- start 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.765+0000 7eff4fa48640 1 -- start start 2023-12-13T23:54:49.767 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 0x7eff4813aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 0x7eff4815b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4813c290 con 0x7eff480714c0 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4813c570 con 0x7eff48143820 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4fa48640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4813c850 con 0x7eff48071970 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4cfe3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 0x7eff4815b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.768 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4cfe3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:34044/0 (socket says 172.21.15.203:34044) 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4cfe3640 1 -- 172.21.15.203:0/1148941613 learned_addr learned my addr 172.21.15.203:0/1148941613 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4d7e4640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 0x7eff4813aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 -- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 msgr2=0x7eff481593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 -- 172.21.15.203:0/1148941613 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 msgr2=0x7eff4813aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 0x7eff4813aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 -- 172.21.15.203:0/1148941613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff44009400 con 0x7eff48143820 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4cfe3640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.769 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4d7e4640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 0x7eff4813aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.766+0000 7eff4dfe5640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 0x7eff4815b870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c700 tx=0x7eff4000cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40003b70 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff40003cf0 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4815b070 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4000aa90 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4813b3e0 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7eff400050d0 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.767+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7eff40012420 con 0x7eff48143820 2023-12-13T23:54:49.770 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.768+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4815b590 con 0x7eff48143820 2023-12-13T23:54:49.771 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.770+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7eff40005380 con 0x7eff48143820 2023-12-13T23:54:49.840 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.842+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e7ce641b-9ed7-4361-a39b-5551b768c71e", "id": 5} v 0) v1 -- 0x7eff48067420 con 0x7eff48143820 2023-12-13T23:54:49.845 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.847+0000 7eff3dffb640 1 -- 172.21.15.203:0/1148941613 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e7ce641b-9ed7-4361-a39b-5551b768c71e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7eff40005520 con 0x7eff48143820 2023-12-13T23:54:49.845 INFO:teuthology.orchestra.run.smithi203.stdout:5 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 msgr2=0x7eff4815b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 0x7eff4815b870 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff4000c700 tx=0x7eff4000cbc0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 shutdown_connections 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff48143820 0x7eff4815b870 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff48071970 0x7eff481593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 --2- 172.21.15.203:0/1148941613 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff480714c0 0x7eff4813aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 >> 172.21.15.203:0/1148941613 conn(0x7eff480b1de0 msgr2=0x7eff4815ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 shutdown_connections 2023-12-13T23:54:49.847 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.848+0000 7eff4fa48640 1 -- 172.21.15.203:0/1148941613 wait complete. 2023-12-13T23:54:49.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7 6 2023-12-13T23:54:49.957 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.959+0000 7f44253a0640 1 Processor -- start 2023-12-13T23:54:49.958 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f44253a0640 1 -- start start 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f44253a0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f44200b2cd0 0x7f44200b3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f44253a0640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44200b3660 con 0x7f442015da90 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f44253a0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44200ade80 con 0x7f44200cf410 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f44253a0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442015e000 con 0x7f44200b2cd0 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.960+0000 7f441ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f44200cf410 0x7f442015af20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:33516/0 (socket says 172.21.15.203:33516) 2023-12-13T23:54:49.959 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f441ffff640 1 -- 172.21.15.203:0/3520138668 learned_addr learned my addr 172.21.15.203:0/3520138668 (peer_addr_for_me v1:172.21.15.203:0/0) 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f4424b9f640 1 --2- 172.21.15.203:0/3520138668 >> v2:172.21.15.124:3300/0 conn(0x7f44200b2cd0 0x7f44200b3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774378067 0 0) 0x7f44200ade80 con 0x7f44200cf410 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4408003520 con 0x7f44200cf410 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754218005 0 0) 0x7f4410001110 con 0x7f442015da90 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44200ade80 con 0x7f442015da90 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f4424b9f640 1 -- 172.21.15.203:0/3520138668 >> v1:172.21.15.203:6789/0 conn(0x7f44200cf410 legacy=0x7f442015af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f4424b9f640 1 -- 172.21.15.203:0/3520138668 >> v1:172.21.15.124:6789/0 conn(0x7f442015da90 legacy=0x7f44200ad770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.961+0000 7f4424b9f640 1 -- 172.21.15.203:0/3520138668 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442015e2e0 con 0x7f44200b2cd0 2023-12-13T23:54:49.960 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f4424b9f640 1 --2- 172.21.15.203:0/3520138668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44200b2cd0 0x7f44200b3120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4418009d90 tx=0x7f441802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120a2c83ebb8be50 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.961 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f441802cd00 con 0x7f44200b2cd0 2023-12-13T23:54:49.961 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f441802ce60 con 0x7f44200b2cd0 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f441effd640 1 -- 172.21.15.203:0/3520138668 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418034660 con 0x7f44200b2cd0 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f44253a0640 1 -- 172.21.15.203:0/3520138668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44200b2cd0 msgr2=0x7f44200b3120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.962+0000 7f44253a0640 1 --2- 172.21.15.203:0/3520138668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44200b2cd0 0x7f44200b3120 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f4418009d90 tx=0x7f441802bf50 comp rx=0 tx=0).stop 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.963+0000 7f44253a0640 1 -- 172.21.15.203:0/3520138668 shutdown_connections 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.963+0000 7f44253a0640 1 --2- 172.21.15.203:0/3520138668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44200b2cd0 0x7f44200b3120 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.963+0000 7f44253a0640 1 -- 172.21.15.203:0/3520138668 >> 172.21.15.203:0/3520138668 conn(0x7f442006b7c0 msgr2=0x7f442006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.963+0000 7f44253a0640 1 -- 172.21.15.203:0/3520138668 shutdown_connections 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.963+0000 7f44253a0640 1 -- 172.21.15.203:0/3520138668 wait complete. 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.964+0000 7f44253a0640 1 Processor -- start 2023-12-13T23:54:49.962 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.964+0000 7f44253a0640 1 -- start start 2023-12-13T23:54:49.964 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 0x7f44201457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.964 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f442015da90 0x7f442006a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420150960 con 0x7f442015da90 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420145d10 con 0x7f44200cf410 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f44253a0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442014fce0 con 0x7f44200b2cd0 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 0x7f44201457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441f7fe640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:48994/0 (socket says 172.21.15.203:48994) 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441f7fe640 1 -- 172.21.15.203:0/2396801886 learned_addr learned my addr 172.21.15.203:0/2396801886 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f4424b9f640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f442015da90 0x7f442006a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 -- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 msgr2=0x7f44200b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 -- 172.21.15.203:0/2396801886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f442015da90 msgr2=0x7f442006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f442015da90 0x7f442006a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:49.965 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 -- 172.21.15.203:0/2396801886 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4418009a70 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441f7fe640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441ffff640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 0x7f44201457d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4414001200 tx=0x7f441400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.965+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414005ce0 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4414005e40 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4414007310 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442014ffc0 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4420150580 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f441401c440 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f441401b3b0 con 0x7f44200b2cd0 2023-12-13T23:54:49.966 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.966+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44200af910 con 0x7f44200b2cd0 2023-12-13T23:54:49.968 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:49.970+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f4414017070 con 0x7f44200b2cd0 2023-12-13T23:54:50.010 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.012+0000 7efe50a37040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-13T23:54:50.016 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.018+0000 7f2fbb616040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-13T23:54:50.040 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.043+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7", "id": 6} v 0) v1 -- 0x7f4420067420 con 0x7f44200b2cd0 2023-12-13T23:54:50.045 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.048+0000 7f441cff9640 1 -- 172.21.15.203:0/2396801886 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f441401c920 con 0x7f44200b2cd0 2023-12-13T23:54:50.046 INFO:teuthology.orchestra.run.smithi203.stdout:6 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.048+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 msgr2=0x7f44201457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 0x7f44201457d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4414001200 tx=0x7f441400c8f0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 shutdown_connections 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f44200cf410 0x7f44200b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f442015da90 0x7f442006a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 --2- 172.21.15.203:0/2396801886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f44200b2cd0 0x7f44201457d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 >> 172.21.15.203:0/2396801886 conn(0x7f442006b7c0 msgr2=0x7f44201584c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:50.047 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 shutdown_connections 2023-12-13T23:54:50.048 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.049+0000 7f44253a0640 1 -- 172.21.15.203:0/2396801886 wait complete. 2023-12-13T23:54:50.057 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 288daefa-555d-4bea-99c3-57ca8b745384 7 2023-12-13T23:54:50.144 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.146+0000 7efe50a37040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-13T23:54:50.156 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.158+0000 7f2fbb616040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-13T23:54:50.156 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.159+0000 7ff97370f640 1 Processor -- start 2023-12-13T23:54:50.157 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.159+0000 7ff97370f640 1 -- start start 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff97370f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff96c06abe0 0x7ff96c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff97370f640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96c161ad0 con 0x7ff96c0af560 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff97370f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff96c160ec0 con 0x7ff96c15d3f0 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff97370f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff96c1611a0 con 0x7ff96c06abe0 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff96c06abe0 0x7ff96c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff96c06abe0 0x7ff96c0af020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:49006/0 (socket says 172.21.15.203:49006) 2023-12-13T23:54:50.158 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 -- 172.21.15.203:0/1427885094 learned_addr learned my addr 172.21.15.203:0/1427885094 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:50.159 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 -- 172.21.15.203:0/1427885094 >> v1:172.21.15.203:6789/0 conn(0x7ff96c15d3f0 legacy=0x7ff96c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.159 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 -- 172.21.15.203:0/1427885094 >> v1:172.21.15.124:6789/0 conn(0x7ff96c0af560 legacy=0x7ff96c0ad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.159 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.160+0000 7ff970caa640 1 -- 172.21.15.203:0/1427885094 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c161480 con 0x7ff96c06abe0 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.161+0000 7ff970caa640 1 --2- 172.21.15.203:0/1427885094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c06abe0 0x7ff96c0af020 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009870 tx=0x7ff95c02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc453c5d99eaff9b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.161+0000 7ff963fff640 1 -- 172.21.15.203:0/1427885094 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c030860 con 0x7ff96c06abe0 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.161+0000 7ff963fff640 1 -- 172.21.15.203:0/1427885094 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff95c0309c0 con 0x7ff96c06abe0 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.161+0000 7ff963fff640 1 -- 172.21.15.203:0/1427885094 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95c030d50 con 0x7ff96c06abe0 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 -- 172.21.15.203:0/1427885094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c06abe0 msgr2=0x7ff96c0af020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 --2- 172.21.15.203:0/1427885094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c06abe0 0x7ff96c0af020 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff95c009870 tx=0x7ff95c02b9a0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 -- 172.21.15.203:0/1427885094 shutdown_connections 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 --2- 172.21.15.203:0/1427885094 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c06abe0 0x7ff96c0af020 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.160 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 -- 172.21.15.203:0/1427885094 >> 172.21.15.203:0/1427885094 conn(0x7ff96c06d3b0 msgr2=0x7ff96c06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:50.161 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 -- 172.21.15.203:0/1427885094 shutdown_connections 2023-12-13T23:54:50.161 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.162+0000 7ff97370f640 1 -- 172.21.15.203:0/1427885094 wait complete. 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.163+0000 7ff97370f640 1 Processor -- start 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.163+0000 7ff97370f640 1 -- start start 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff96c15d3f0 0x7ff96c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c154680 con 0x7ff96c15d3f0 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1535e0 con 0x7ff96c0af560 2023-12-13T23:54:50.163 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1538c0 con 0x7ff96c06abe0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff970caa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.203:34054/0 (socket says 172.21.15.203:34054) 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff970caa640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.203:49008/0 (socket says 172.21.15.203:49008) 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 -- 172.21.15.203:0/376401221 learned_addr learned my addr 172.21.15.203:0/376401221 (peer_addr_for_me v2:172.21.15.203:0/0) 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff971cac640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff96c15d3f0 0x7ff96c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 -- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 msgr2=0x7ff96c15da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 -- 172.21.15.203:0/376401221 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff96c15d3f0 msgr2=0x7ff96c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff96c15d3f0 0x7ff96c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 -- 172.21.15.203:0/376401221 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c0094d0 con 0x7ff96c06abe0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff970caa640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff9714ab640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff96800c400 tx=0x7ff96800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:50.164 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9680052d0 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff968005430 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96c153ba0 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.164+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968003830 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.165+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96c154160 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.165+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7ff968003d10 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.165+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7ff968012a80 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.166+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96c15cc30 con 0x7ff96c06abe0 2023-12-13T23:54:50.165 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.167+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7ff968017070 con 0x7ff96c06abe0 2023-12-13T23:54:50.237 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.239+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd new", "uuid": "288daefa-555d-4bea-99c3-57ca8b745384", "id": 7} v 0) v1 -- 0x7ff96c067420 con 0x7ff96c06abe0 2023-12-13T23:54:50.243 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.245+0000 7ff961ffb640 1 -- 172.21.15.203:0/376401221 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "288daefa-555d-4bea-99c3-57ca8b745384", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff9680124e0 con 0x7ff96c06abe0 2023-12-13T23:54:50.243 INFO:teuthology.orchestra.run.smithi203.stdout:7 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 msgr2=0x7ff96c1f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff96800c400 tx=0x7ff96800c8c0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 shutdown_connections 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff96c0af560 0x7ff96c15da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff96c15d3f0 0x7ff96c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 --2- 172.21.15.203:0/376401221 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff96c06abe0 0x7ff96c1f2460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 >> 172.21.15.203:0/376401221 conn(0x7ff96c06d3b0 msgr2=0x7ff96c06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 shutdown_connections 2023-12-13T23:54:50.244 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:54:50.246+0000 7ff97370f640 1 -- 172.21.15.203:0/376401221 wait complete. 2023-12-13T23:54:50.254 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-13T23:54:50.254 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-12-13T23:54:50.256 INFO:tasks.ceph.osd.0:Started 2023-12-13T23:54:50.256 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-13T23:54:50.256 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-12-13T23:54:50.258 INFO:tasks.ceph.osd.1:Started 2023-12-13T23:54:50.258 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-13T23:54:50.258 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-12-13T23:54:50.260 INFO:tasks.ceph.osd.2:Started 2023-12-13T23:54:50.260 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-13T23:54:50.260 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-12-13T23:54:50.262 INFO:tasks.ceph.osd.3:Started 2023-12-13T23:54:50.262 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-13T23:54:50.263 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-12-13T23:54:50.265 INFO:tasks.ceph.osd.4:Started 2023-12-13T23:54:50.265 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-13T23:54:50.265 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-12-13T23:54:50.267 INFO:tasks.ceph.osd.5:Started 2023-12-13T23:54:50.267 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-13T23:54:50.267 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-12-13T23:54:50.269 INFO:tasks.ceph.osd.6:Started 2023-12-13T23:54:50.269 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-13T23:54:50.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-12-13T23:54:50.270 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.272+0000 7f2fbb616040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-13T23:54:50.271 INFO:tasks.ceph.osd.7:Started 2023-12-13T23:54:50.272 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:54:50.272 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.274+0000 7efe50a37040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-13T23:54:50.335 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.337+0000 7efe50a37040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-13T23:54:50.340 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.342+0000 7f2fbb616040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-13T23:54:50.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.403+0000 7f09f43e3640 1 Processor -- start 2023-12-13T23:54:50.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.403+0000 7f09f43e3640 1 -- start start 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f43e3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f09ec143c50 0x7f09ec0721d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f43e3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 0x7f09ec0b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f43e3640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09ec0b3d40 con 0x7f09ec0b4440 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f43e3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09ec0b4020 con 0x7f09ec143c50 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f43e3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec0b4e00 con 0x7f09ec0b3760 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f197e640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f09ec0b4440 0x7f09ec0b3050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:58778/0 (socket says 172.21.15.124:58778) 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f197e640 1 -- 172.21.15.124:0/391336671 learned_addr learned my addr 172.21.15.124:0/391336671 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f217f640 1 --2- 172.21.15.124:0/391336671 >> v2:172.21.15.203:3300/0 conn(0x7f09ec143c50 0x7f09ec0721d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.404+0000 7f09f2980640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 0x7f09ec0b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f117d640 1 -- 172.21.15.124:0/391336671 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2174972022 0 0) 0x7f09ec0b3d40 con 0x7f09ec0b4440 2023-12-13T23:54:50.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f117d640 1 -- 172.21.15.124:0/391336671 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09d8003520 con 0x7f09ec0b4440 2023-12-13T23:54:50.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f217f640 1 -- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 msgr2=0x7f09ec0b48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:50.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f217f640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 0x7f09ec0b48c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f217f640 1 -- 172.21.15.124:0/391336671 >> v1:172.21.15.124:6789/0 conn(0x7f09ec0b4440 legacy=0x7f09ec0b3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f217f640 1 -- 172.21.15.124:0/391336671 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec0716f0 con 0x7f09ec143c50 2023-12-13T23:54:50.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.405+0000 7f09f2980640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 0x7f09ec0b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f217f640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec143c50 0x7f09ec0721d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0097a0 tx=0x7f09dc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0203497043bce36 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f117d640 1 -- 172.21.15.124:0/391336671 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031780 con 0x7f09ec143c50 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f117d640 1 -- 172.21.15.124:0/391336671 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc0318e0 con 0x7f09ec143c50 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f117d640 1 -- 172.21.15.124:0/391336671 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc031c70 con 0x7f09ec143c50 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 -- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec143c50 msgr2=0x7f09ec0721d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec143c50 0x7f09ec0721d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f09dc0097a0 tx=0x7f09dc02b820 comp rx=0 tx=0).stop 2023-12-13T23:54:50.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 -- 172.21.15.124:0/391336671 shutdown_connections 2023-12-13T23:54:50.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec0b3760 0x7f09ec0b48c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 --2- 172.21.15.124:0/391336671 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec143c50 0x7f09ec0721d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 -- 172.21.15.124:0/391336671 >> 172.21.15.124:0/391336671 conn(0x7f09ec0b1d50 msgr2=0x7f09ec0b2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:50.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 -- 172.21.15.124:0/391336671 shutdown_connections 2023-12-13T23:54:50.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.406+0000 7f09f43e3640 1 -- 172.21.15.124:0/391336671 wait complete. 2023-12-13T23:54:50.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.408+0000 7f09f43e3640 1 Processor -- start 2023-12-13T23:54:50.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.408+0000 7f09f43e3640 1 -- start start 2023-12-13T23:54:50.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 0x7f09ec136ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 0x7f09ec0a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec153340 con 0x7f09ec0b4440 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec1381c0 con 0x7f09ec0b3760 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f197e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f43e3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09ec137920 con 0x7f09ec143c50 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f197e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:34426/0 (socket says 172.21.15.124:34426) 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f197e640 1 -- 172.21.15.124:0/3435243922 learned_addr learned my addr 172.21.15.124:0/3435243922 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 0x7f09ec136ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f2980640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 0x7f09ec0a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 -- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 msgr2=0x7f09ec0a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 0x7f09ec0a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 -- 172.21.15.124:0/3435243922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 msgr2=0x7f09ec0a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 -- 172.21.15.124:0/3435243922 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4009a70 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f2980640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 0x7f09ec0a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f197e640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09f217f640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 0x7f09ec136ed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f09dc033000 tx=0x7f09dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.409+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc032820 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09dc0329a0 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09dc009400 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09dc032d50 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09ec137ee0 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f09dc045780 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.410+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f09dc03b3f0 con 0x7f09ec0b3760 2023-12-13T23:54:50.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.411+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09ec067420 con 0x7f09ec0b3760 2023-12-13T23:54:50.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.413+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f09dc04e3f0 con 0x7f09ec0b3760 2023-12-13T23:54:50.445 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.447+0000 7f2fbb616040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-13T23:54:50.446 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.448+0000 7efe50a37040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-13T23:54:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.473+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09ec0b2a90 con 0x7f09ec0b3760 2023-12-13T23:54:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.473+0000 7f09e2ffd640 1 -- 172.21.15.124:0/3435243922 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f09dc045a30 con 0x7f09ec0b3760 2023-12-13T23:54:50.472 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:54:50.472 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":10,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:54:50.241560+0000","last_up_change":"0.000000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 msgr2=0x7f09ec136ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 0x7f09ec136ed0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f09dc033000 tx=0x7f09dc0046c0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 shutdown_connections 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09ec143c50 0x7f09ec0a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09ec0b4440 0x7f09ec0a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 --2- 172.21.15.124:0/3435243922 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f09ec0b3760 0x7f09ec136ed0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 >> 172.21.15.124:0/3435243922 conn(0x7f09ec0b1d50 msgr2=0x7f09ec06ef10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 shutdown_connections 2023-12-13T23:54:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:54:50.474+0000 7f09f43e3640 1 -- 172.21.15.124:0/3435243922 wait complete. 2023-12-13T23:54:50.483 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-13T23:54:50.483 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-13T23:54:50.538 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.540+0000 7f2fbb616040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-13T23:54:50.544 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.547+0000 7efe50a37040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-13T23:54:50.678 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.681+0000 7f2fbb616040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-13T23:54:50.686 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.688+0000 7efe50a37040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-13T23:54:50.720 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.722+0000 7f2fbb616040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-13T23:54:50.727 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.730+0000 7efe50a37040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-13T23:54:50.857 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:50.859+0000 7f2fbb616040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-13T23:54:50.865 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-12-13T23:54:50.867+0000 7f2c6b5bc500 -1 Falling back to public interface 2023-12-13T23:54:50.868 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:50.870+0000 7efe50a37040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-13T23:54:50.872 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-12-13T23:54:50.874+0000 7efc7661c500 -1 Falling back to public interface 2023-12-13T23:54:50.872 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-12-13T23:54:50.874+0000 7f023be0e500 -1 Falling back to public interface 2023-12-13T23:54:50.872 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-12-13T23:54:50.874+0000 7efc74f06640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2023-12-13T23:54:50.873 INFO:tasks.ceph.osd.1.smithi124.stderr:2023-12-13T23:54:50.874+0000 7ff887aa9500 -1 Falling back to public interface 2023-12-13T23:54:50.877 INFO:tasks.ceph.osd.2.smithi124.stderr:2023-12-13T23:54:50.879+0000 7f6eb2869500 -1 Falling back to public interface 2023-12-13T23:54:50.880 INFO:tasks.ceph.osd.0.smithi124.stderr:2023-12-13T23:54:50.882+0000 7fa552394500 -1 Falling back to public interface 2023-12-13T23:54:50.881 INFO:tasks.ceph.osd.3.smithi124.stderr:2023-12-13T23:54:50.882+0000 7f45e3d07500 -1 Falling back to public interface 2023-12-13T23:54:50.881 INFO:tasks.ceph.osd.3.smithi124.stderr:2023-12-13T23:54:50.882+0000 7f45e2cb0640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-12-13T23:54:50.885 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-12-13T23:54:50.887+0000 7f244e212500 -1 Falling back to public interface 2023-12-13T23:54:51.287 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-13T23:54:51.377 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.379+0000 7f27a30b6640 1 Processor -- start 2023-12-13T23:54:51.377 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.379+0000 7f27a30b6640 1 -- start start 2023-12-13T23:54:51.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a30b6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f279c0d3620 0x7f279c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:51.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a30b6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0d1680 con 0x7f279c15d570 2023-12-13T23:54:51.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a30b6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f279c0d0d90 con 0x7f279c15d190 2023-12-13T23:54:51.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a30b6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f279c0d2220 con 0x7f279c0d3620 2023-12-13T23:54:51.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f279bfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f279c15d570 0x7f279c0d2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:37298/0 (socket says 172.21.15.124:37298) 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f279bfff640 1 -- 172.21.15.124:0/3001741498 learned_addr learned my addr 172.21.15.124:0/3001741498 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a1653640 1 --2- 172.21.15.124:0/3001741498 >> v2:172.21.15.203:3300/0 conn(0x7f279c0d3620 0x7f279c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f279b7fe640 1 -- 172.21.15.124:0/3001741498 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100190681 0 0) 0x7f279c0d1680 con 0x7f279c15d570 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f279b7fe640 1 -- 172.21.15.124:0/3001741498 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c003520 con 0x7f279c15d570 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a1653640 1 -- 172.21.15.124:0/3001741498 >> v1:172.21.15.203:6789/0 conn(0x7f279c15d190 legacy=0x7f279c0d1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a1653640 1 -- 172.21.15.124:0/3001741498 >> v1:172.21.15.124:6789/0 conn(0x7f279c15d570 legacy=0x7f279c0d2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.380+0000 7f27a1653640 1 -- 172.21.15.124:0/3001741498 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f279c0b4060 con 0x7f279c0d3620 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.381+0000 7f27a1653640 1 --2- 172.21.15.124:0/3001741498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c0d3620 0x7f279c0b32f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2794009d90 tx=0x7f279402bfd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15b2883d2152f08a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.381+0000 7f279b7fe640 1 -- 172.21.15.124:0/3001741498 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f279402cd80 con 0x7f279c0d3620 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.381+0000 7f279b7fe640 1 -- 172.21.15.124:0/3001741498 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2794034440 con 0x7f279c0d3620 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.381+0000 7f279b7fe640 1 -- 172.21.15.124:0/3001741498 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27940347b0 con 0x7f279c0d3620 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 -- 172.21.15.124:0/3001741498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c0d3620 msgr2=0x7f279c0b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 --2- 172.21.15.124:0/3001741498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c0d3620 0x7f279c0b32f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2794009d90 tx=0x7f279402bfd0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 -- 172.21.15.124:0/3001741498 shutdown_connections 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 --2- 172.21.15.124:0/3001741498 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c0d3620 0x7f279c0b32f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 -- 172.21.15.124:0/3001741498 >> 172.21.15.124:0/3001741498 conn(0x7f279c08cbe0 msgr2=0x7f279c09c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 -- 172.21.15.124:0/3001741498 shutdown_connections 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.382+0000 7f27a30b6640 1 -- 172.21.15.124:0/3001741498 wait complete. 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.383+0000 7f27a30b6640 1 Processor -- start 2023-12-13T23:54:51.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.383+0000 7f27a30b6640 1 -- start start 2023-12-13T23:54:51.383 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f279c0d3620 0x7f279c1603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 0x7f279c155fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c1f3cc0 con 0x7f279c15d190 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279c0b7310 con 0x7f279c15d570 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a30b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f279c0b75f0 con 0x7f279c0d3620 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:47190/0 (socket says 172.21.15.124:47190) 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 -- 172.21.15.124:0/352224215 learned_addr learned my addr 172.21.15.124:0/352224215 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a1653640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 0x7f279c155fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 -- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f279c0d3620 msgr2=0x7f279c1603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a0e52640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f279c0d3620 0x7f279c1603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f279c0d3620 0x7f279c1603b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 -- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 msgr2=0x7f279c155fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 0x7f279c155fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 -- 172.21.15.124:0/352224215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2794009a70 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f27a1653640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 0x7f279c155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.384+0000 7f279bfff640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f278800c410 tx=0x7f278800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2788005ce0 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2788005e40 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f279c0b78d0 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27880072f0 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279c0b7e90 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 799+0+0 (secure 0 0 0) 0x7f278801c440 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.385+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f278801b3b0 con 0x7f279c15d190 2023-12-13T23:54:51.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.386+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f279c069320 con 0x7f279c15d190 2023-12-13T23:54:51.386 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.388+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72904 (secure 0 0 0) 0x7f2788017070 con 0x7f279c15d190 2023-12-13T23:54:51.469 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.470+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f279c0c70e0 con 0x7f279c15d190 2023-12-13T23:54:51.469 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.471+0000 7f27997fa640 1 -- 172.21.15.124:0/352224215 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f27880124b0 con 0x7f279c15d190 2023-12-13T23:54:51.469 INFO:teuthology.misc.health.smithi124.stdout: 2023-12-13T23:54:51.469 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":10,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:54:50.241560+0000","last_up_change":"0.000000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 msgr2=0x7f279c1f33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f278800c410 tx=0x7f278800c8d0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 shutdown_connections 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f279c15d570 0x7f279c155fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f279c15d190 0x7f279c1f33e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 --2- 172.21.15.124:0/352224215 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f279c0d3620 0x7f279c1603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 >> 172.21.15.124:0/352224215 conn(0x7f279c08cbe0 msgr2=0x7f279c0b5110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 shutdown_connections 2023-12-13T23:54:51.472 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:51.472+0000 7f27a30b6640 1 -- 172.21.15.124:0/352224215 wait complete. 2023-12-13T23:54:51.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-13T23:54:51.655 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:51.658+0000 7f2fbb616040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-13T23:54:51.696 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:51.698+0000 7efe50a37040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-13T23:54:51.927 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:51.929+0000 7f2fbb616040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-13T23:54:51.970 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:51.972+0000 7efe50a37040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-13T23:54:51.979 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:51.981+0000 7f2fbb616040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-13T23:54:52.023 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:52.025+0000 7efe50a37040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-13T23:54:52.082 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:52.084+0000 7f2fbb616040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-13T23:54:52.125 INFO:tasks.ceph.mgr.y.smithi124.stderr:2023-12-13T23:54:52.127+0000 7efe50a37040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-13T23:54:53.109 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:53.111+0000 7f2f5c4d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-13T23:54:53.109 INFO:tasks.ceph.mgr.x.smithi203.stderr:2023-12-13T23:54:53.112+0000 7f2f5c4d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-13T23:54:54.338 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-12-13T23:54:54.341+0000 7f2c6b5bc500 -1 osd.7 0 log_to_monitors true 2023-12-13T23:54:54.389 INFO:tasks.ceph.osd.3.smithi124.stderr:2023-12-13T23:54:54.391+0000 7f45e3d07500 -1 osd.3 0 log_to_monitors true 2023-12-13T23:54:54.389 INFO:tasks.ceph.osd.2.smithi124.stderr:2023-12-13T23:54:54.391+0000 7f6eb2869500 -1 osd.2 0 log_to_monitors true 2023-12-13T23:54:54.401 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-12-13T23:54:54.403+0000 7f023be0e500 -1 osd.6 0 log_to_monitors true 2023-12-13T23:54:54.405 INFO:tasks.ceph.osd.1.smithi124.stderr:2023-12-13T23:54:54.407+0000 7ff887aa9500 -1 osd.1 0 log_to_monitors true 2023-12-13T23:54:54.410 INFO:tasks.ceph.osd.0.smithi124.stderr:2023-12-13T23:54:54.412+0000 7fa552394500 -1 osd.0 0 log_to_monitors true 2023-12-13T23:54:54.412 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-12-13T23:54:54.414+0000 7efc7661c500 -1 osd.4 0 log_to_monitors true 2023-12-13T23:54:54.417 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-12-13T23:54:54.419+0000 7f244e212500 -1 osd.5 0 log_to_monitors true 2023-12-13T23:54:58.098 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-12-13T23:54:58.100+0000 7efc716ca640 -1 osd.4 0 waiting for initial osdmap 2023-12-13T23:54:58.114 INFO:tasks.ceph.osd.4.smithi203.stderr:2023-12-13T23:54:58.116+0000 7efc69d03640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.119 INFO:tasks.ceph.osd.0.smithi124.stderr:2023-12-13T23:54:58.121+0000 7fa54db01640 -1 osd.0 0 waiting for initial osdmap 2023-12-13T23:54:58.125 INFO:tasks.ceph.osd.2.smithi124.stderr:2023-12-13T23:54:58.127+0000 7f6eadfd4640 -1 osd.2 0 waiting for initial osdmap 2023-12-13T23:54:58.135 INFO:tasks.ceph.osd.0.smithi124.stderr:2023-12-13T23:54:58.137+0000 7fa54613a640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.137 INFO:tasks.ceph.osd.2.smithi124.stderr:2023-12-13T23:54:58.140+0000 7f6ea660d640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.141 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-12-13T23:54:58.143+0000 7f24493db640 -1 osd.5 0 waiting for initial osdmap 2023-12-13T23:54:58.162 INFO:tasks.ceph.osd.5.smithi203.stderr:2023-12-13T23:54:58.164+0000 7f2441a14640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.197 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-12-13T23:54:58.199+0000 7f2c6855f640 -1 osd.7 0 waiting for initial osdmap 2023-12-13T23:54:58.206 INFO:tasks.ceph.osd.7.smithi203.stderr:2023-12-13T23:54:58.209+0000 7f2c5f360640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.216 INFO:tasks.ceph.osd.1.smithi124.stderr:2023-12-13T23:54:58.218+0000 7ff883214640 -1 osd.1 0 waiting for initial osdmap 2023-12-13T23:54:58.227 INFO:tasks.ceph.osd.1.smithi124.stderr:2023-12-13T23:54:58.229+0000 7ff87b84d640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.241 INFO:tasks.ceph.osd.3.smithi124.stderr:2023-12-13T23:54:58.244+0000 7f45df474640 -1 osd.3 0 waiting for initial osdmap 2023-12-13T23:54:58.246 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-12-13T23:54:58.248+0000 7f0236f11640 -1 osd.6 0 waiting for initial osdmap 2023-12-13T23:54:58.252 INFO:tasks.ceph.osd.3.smithi124.stderr:2023-12-13T23:54:58.255+0000 7f45d7aad640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.258 INFO:tasks.ceph.osd.6.smithi203.stderr:2023-12-13T23:54:58.261+0000 7f022f54a640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-13T23:54:58.289 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-13T23:54:58.377 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.379+0000 7f4685fc0640 1 Processor -- start 2023-12-13T23:54:58.378 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.379+0000 7f4685fc0640 1 -- start start 2023-12-13T23:54:58.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f4685fc0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f468006ea10 0x7f46800cddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f4685fc0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f46800716e0 0x7f4680071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f4685fc0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468006fd40 con 0x7f46800cf2a0 2023-12-13T23:54:58.379 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f4685fc0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468006eea0 con 0x7f46800716e0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f4685fc0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468015b870 con 0x7f468006ea10 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f467ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f46800716e0 0x7f4680071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f467ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f46800716e0 0x7f4680071b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:47232/0 (socket says 172.21.15.124:47232) 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f467ffff640 1 -- 172.21.15.124:0/3641791865 learned_addr learned my addr 172.21.15.124:0/3641791865 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.380+0000 7f467effd640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f468006ea10 0x7f46800cddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467ffff640 1 -- 172.21.15.124:0/3641791865 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f468006ea10 msgr2=0x7f46800cddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467ffff640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f468006ea10 0x7f46800cddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467ffff640 1 -- 172.21.15.124:0/3641791865 >> v1:172.21.15.203:6789/0 conn(0x7f46800cf2a0 legacy=0x7f46800cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467ffff640 1 -- 172.21.15.124:0/3641791865 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46800cf040 con 0x7f46800716e0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467ffff640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46800716e0 0x7f4680071b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e50 tx=0x7f467802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4bf729b77311a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.381+0000 7f467e7fc640 1 -- 172.21.15.124:0/3641791865 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467802ca70 con 0x7f46800716e0 2023-12-13T23:54:58.380 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f467e7fc640 1 -- 172.21.15.124:0/3641791865 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467802cbd0 con 0x7f46800716e0 2023-12-13T23:54:58.381 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f467e7fc640 1 -- 172.21.15.124:0/3641791865 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46780343f0 con 0x7f46800716e0 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 -- 172.21.15.124:0/3641791865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46800716e0 msgr2=0x7f4680071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46800716e0 0x7f4680071b50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e50 tx=0x7f467802bed0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 -- 172.21.15.124:0/3641791865 shutdown_connections 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f468006ea10 0x7f46800cddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 --2- 172.21.15.124:0/3641791865 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46800716e0 0x7f4680071b50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 -- 172.21.15.124:0/3641791865 >> 172.21.15.124:0/3641791865 conn(0x7f46800b1c70 msgr2=0x7f46800b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 -- 172.21.15.124:0/3641791865 shutdown_connections 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.382+0000 7f4685fc0640 1 -- 172.21.15.124:0/3641791865 wait complete. 2023-12-13T23:54:58.384 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.383+0000 7f4685fc0640 1 Processor -- start 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.383+0000 7f4685fc0640 1 -- start start 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46800716e0 0x7f46800ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 0x7f46801f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468013fb70 con 0x7f468006ea10 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468015ca60 con 0x7f46800cf2a0 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f4685fc0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468015d300 con 0x7f46800716e0 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:47248/0 (socket says 172.21.15.124:47248) 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467f7fe640 1 -- 172.21.15.124:0/2119446151 learned_addr learned my addr 172.21.15.124:0/2119446151 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467ffff640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 0x7f46801f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467effd640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46800716e0 0x7f46800ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.385 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467f7fe640 1 -- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46800716e0 msgr2=0x7f46800ad2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.386 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.384+0000 7f467f7fe640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46800716e0 0x7f46800ad2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f467f7fe640 1 -- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 msgr2=0x7f46801f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f467f7fe640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 0x7f46801f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f467f7fe640 1 -- 172.21.15.124:0/2119446151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009b30 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f467ffff640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 0x7f46801f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f467f7fe640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f467000c450 tx=0x7f467000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46700052d0 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4670005430 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46800adcb0 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4670012870 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.385+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468015cf90 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.386+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468015d150 con 0x7f468006ea10 2023-12-13T23:54:58.390 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.386+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 4) v1 ==== 89314+0+0 (secure 0 0 0) 0x7f46700129d0 con 0x7f468006ea10 2023-12-13T23:54:58.391 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.387+0000 7f4667fff640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 0x7f466c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:54:58.391 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.387+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f46700885e0 con 0x7f468006ea10 2023-12-13T23:54:58.391 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.387+0000 7f467effd640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 0x7f466c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:54:58.392 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.387+0000 7f467effd640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 0x7f466c06d5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f46801f6100 tx=0x7f4674008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:54:58.392 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.391+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4670054210 con 0x7f468006ea10 2023-12-13T23:54:58.558 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.560+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4680069320 con 0x7f468006ea10 2023-12-13T23:54:58.559 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.561+0000 7f4667fff640 1 -- 172.21.15.124:0/2119446151 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7058 (secure 0 0 0) 0x7f4670057830 con 0x7f468006ea10 2023-12-13T23:54:58.559 INFO:teuthology.misc.health.smithi124.stdout: 2023-12-13T23:54:58.559 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":12,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:54:56.106814+0000","last_up_change":"0.000000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 msgr2=0x7f466c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 0x7f466c06d5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f46801f6100 tx=0x7f4674008040 comp rx=0 tx=0).stop 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 msgr2=0x7f468013c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f467000c450 tx=0x7f467000c910 comp rx=0 tx=0).stop 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 shutdown_connections 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f466c06b130 0x7f466c06d5f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.561 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f46800cf2a0 0x7f46801f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.562 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f46800716e0 0x7f46800ad2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.562 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 --2- 172.21.15.124:0/2119446151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f468006ea10 0x7f468013c900 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:54:58.562 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 >> 172.21.15.124:0/2119446151 conn(0x7f46800b1c70 msgr2=0x7f4680143490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:54:58.562 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 shutdown_connections 2023-12-13T23:54:58.562 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:54:58.563+0000 7f4685fc0640 1 -- 172.21.15.124:0/2119446151 wait complete. 2023-12-13T23:54:58.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-13T23:55:05.377 DEBUG:teuthology.orchestra.run.smithi124:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:05.473 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.475+0000 7f1fef6f5640 1 Processor -- start 2023-12-13T23:55:05.473 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.475+0000 7f1fef6f5640 1 -- start start 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fef6f5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1fe813d270 0x7f1fe813f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fef6f5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1fe813fd20 0x7f1fe815a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fef6f5640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe8140170 con 0x7f1fe813ce90 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fef6f5640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80b4630 con 0x7f1fe813d270 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fef6f5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80b3d60 con 0x7f1fe813fd20 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fecc90640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1fe813d270 0x7f1fe813f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fed491640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f1fe813ce90 0x7f1fe80b4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36870/0 (socket says 172.21.15.124:36870) 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.476+0000 7f1fed491640 1 -- 172.21.15.124:0/2385436796 learned_addr learned my addr 172.21.15.124:0/2385436796 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fecc90640 1 -- 172.21.15.124:0/2385436796 >> v2:172.21.15.203:3300/0 conn(0x7f1fe813fd20 msgr2=0x7f1fe815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fecc90640 1 --2- 172.21.15.124:0/2385436796 >> v2:172.21.15.203:3300/0 conn(0x7f1fe813fd20 0x7f1fe815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fecc90640 1 -- 172.21.15.124:0/2385436796 >> v1:172.21.15.124:6789/0 conn(0x7f1fe813ce90 legacy=0x7f1fe80b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fecc90640 1 -- 172.21.15.124:0/2385436796 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80b28c0 con 0x7f1fe813d270 2023-12-13T23:55:05.475 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fdffff640 1 -- 172.21.15.124:0/2385436796 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414015168 0 0) 0x7f1fe4001490 con 0x7f1fe813ce90 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fecc90640 1 --2- 172.21.15.124:0/2385436796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813d270 0x7f1fe813f7e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80b1c80 tx=0x7f1fd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e3cb2378cd4383 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fdffff640 1 -- 172.21.15.124:0/2385436796 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8031830 con 0x7f1fe813d270 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fdffff640 1 -- 172.21.15.124:0/2385436796 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8031990 con 0x7f1fe813d270 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.477+0000 7f1fdffff640 1 -- 172.21.15.124:0/2385436796 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8031d20 con 0x7f1fe813d270 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 -- 172.21.15.124:0/2385436796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813d270 msgr2=0x7f1fe813f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/2385436796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813d270 0x7f1fe813f7e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80b1c80 tx=0x7f1fd802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 -- 172.21.15.124:0/2385436796 shutdown_connections 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/2385436796 >> v2:172.21.15.203:3300/0 conn(0x7f1fe813fd20 0x7f1fe815a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/2385436796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813d270 0x7f1fe813f7e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.477 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 -- 172.21.15.124:0/2385436796 >> 172.21.15.124:0/2385436796 conn(0x7f1fe80b0c60 msgr2=0x7f1fe80b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:05.478 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 -- 172.21.15.124:0/2385436796 shutdown_connections 2023-12-13T23:55:05.478 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.478+0000 7f1fef6f5640 1 -- 172.21.15.124:0/2385436796 wait complete. 2023-12-13T23:55:05.480 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.479+0000 7f1fef6f5640 1 Processor -- start 2023-12-13T23:55:05.480 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.479+0000 7f1fef6f5640 1 -- start start 2023-12-13T23:55:05.480 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.480 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1fe813d270 0x7f1fe80afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe813fd20 0x7f1fe80b0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe8152700 con 0x7f1fe813ce90 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe813c6b0 con 0x7f1fe813fd20 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fef6f5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe813c990 con 0x7f1fe813d270 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42048/0 (socket says 172.21.15.124:42048) 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 -- 172.21.15.124:0/561383423 learned_addr learned my addr 172.21.15.124:0/561383423 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fedc92640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe813fd20 0x7f1fe80b0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fecc90640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1fe813d270 0x7f1fe80afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 -- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1fe813d270 msgr2=0x7f1fe80afbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1fe813d270 0x7f1fe80afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 -- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe813fd20 msgr2=0x7f1fe80b0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.481 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe813fd20 0x7f1fe80b0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.480+0000 7f1fed491640 1 -- 172.21.15.124:0/561383423 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd8009400 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fed491640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c890 tx=0x7f1fe400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400a640 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe400a7a0 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400ab30 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe8152950 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.481+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe813b570 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.482+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe8152e70 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.482+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1fe400ac90 con 0x7f1fe813ce90 2023-12-13T23:55:05.482 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.483+0000 7f1fddffb640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 0x7f1fcc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.483 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.483+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1fe40889f0 con 0x7f1fe813ce90 2023-12-13T23:55:05.483 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.483+0000 7f1fecc90640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 0x7f1fcc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.483 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.483+0000 7f1fecc90640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 0x7f1fcc06d680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8036000 tx=0x7f1fd8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.485 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.487+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1fe8152e70 con 0x7f1fe813ce90 2023-12-13T23:55:05.654 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.656+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe8140170 con 0x7f1fe813ce90 2023-12-13T23:55:05.655 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.657+0000 7f1fddffb640 1 -- 172.21.15.124:0/561383423 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12911 (secure 0 0 0) 0x7f1fe8140170 con 0x7f1fe813ce90 2023-12-13T23:55:05.655 INFO:teuthology.misc.health.smithi124.stdout: 2023-12-13T23:55:05.655 INFO:teuthology.misc.health.smithi124.stdout:{"epoch":17,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:03.145969+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:05.657 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 msgr2=0x7f1fcc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 0x7f1fcc06d680 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8036000 tx=0x7f1fd8008030 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 msgr2=0x7f1fe813b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400c890 tx=0x7f1fe400cd50 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 shutdown_connections 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fcc06b1c0 0x7f1fcc06d680 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1fe813fd20 0x7f1fe80b0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1fe813d270 0x7f1fe80afbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 --2- 172.21.15.124:0/561383423 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fe813ce90 0x7f1fe813b030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 >> 172.21.15.124:0/561383423 conn(0x7f1fe80b0c60 msgr2=0x7f1fe80af170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 shutdown_connections 2023-12-13T23:55:05.658 INFO:teuthology.misc.health.smithi124.stderr:2023-12-13T23:55:05.659+0000 7f1fef6f5640 1 -- 172.21.15.124:0/561383423 wait complete. 2023-12-13T23:55:05.667 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-13T23:55:05.667 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-13T23:55:05.668 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-13T23:55:05.668 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-13T23:55:05.670 INFO:tasks.ceph.mds.a:Started 2023-12-13T23:55:05.670 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-13T23:55:05.670 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-13T23:55:05.672 INFO:tasks.ceph.mds.c:Started 2023-12-13T23:55:05.672 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-13T23:55:05.672 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-13T23:55:05.675 INFO:tasks.ceph.mds.b:Started 2023-12-13T23:55:05.675 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-13T23:55:05.675 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-13T23:55:05.678 INFO:tasks.ceph.mds.d:Started 2023-12-13T23:55:05.678 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-12-13T23:55:05.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:05.746 INFO:tasks.ceph.mds.a.smithi124.stdout:starting mds.a at 2023-12-13T23:55:05.746 INFO:tasks.ceph.mds.c.smithi124.stdout:starting mds.c at 2023-12-13T23:55:05.762 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2023-12-13T23:55:05.762 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2023-12-13T23:55:05.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.786+0000 7fd7d3f39640 1 Processor -- start 2023-12-13T23:55:05.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.786+0000 7fd7d3f39640 1 -- start start 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d3f39640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd7cc15d330 0x7fd7cc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d3f39640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc0b2b10 con 0x7fd7cc0d3900 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d3f39640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc0d1590 con 0x7fd7cc0d2910 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d3f39640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc0d1870 con 0x7fd7cc15d330 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd7cc15d330 0x7fd7cc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd7cc15d330 0x7fd7cc0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42098/0 (socket says 172.21.15.124:42098) 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 -- 172.21.15.124:0/464504437 learned_addr learned my addr 172.21.15.124:0/464504437 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d24d6640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd7cc0d3900 0x7fd7cc0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36896/0 (socket says 172.21.15.124:36896) 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 -- 172.21.15.124:0/464504437 >> v1:172.21.15.203:6789/0 conn(0x7fd7cc0d2910 legacy=0x7fd7cc0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 -- 172.21.15.124:0/464504437 >> v1:172.21.15.124:6789/0 conn(0x7fd7cc0d3900 legacy=0x7fd7cc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d1cd5640 1 -- 172.21.15.124:0/464504437 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc0d1b50 con 0x7fd7cc15d330 2023-12-13T23:55:05.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.787+0000 7fd7d0cd3640 1 -- 172.21.15.124:0/464504437 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871638486 0 0) 0x7fd7cc0b2b10 con 0x7fd7cc0d3900 2023-12-13T23:55:05.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.788+0000 7fd7d1cd5640 1 --2- 172.21.15.124:0/464504437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc15d330 0x7fd7cc0b8b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009870 tx=0x7fd7c802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1327f1f87a5e70a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.788+0000 7fd7d0cd3640 1 -- 172.21.15.124:0/464504437 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8030860 con 0x7fd7cc15d330 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.788+0000 7fd7d0cd3640 1 -- 172.21.15.124:0/464504437 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c80309c0 con 0x7fd7cc15d330 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.788+0000 7fd7d0cd3640 1 -- 172.21.15.124:0/464504437 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8030d50 con 0x7fd7cc15d330 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 -- 172.21.15.124:0/464504437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc15d330 msgr2=0x7fd7cc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/464504437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc15d330 0x7fd7cc0b8b80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8009870 tx=0x7fd7c802b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 -- 172.21.15.124:0/464504437 shutdown_connections 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/464504437 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc15d330 0x7fd7cc0b8b80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 -- 172.21.15.124:0/464504437 >> 172.21.15.124:0/464504437 conn(0x7fd7cc072020 msgr2=0x7fd7cc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 -- 172.21.15.124:0/464504437 shutdown_connections 2023-12-13T23:55:05.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.789+0000 7fd7d3f39640 1 -- 172.21.15.124:0/464504437 wait complete. 2023-12-13T23:55:05.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.790+0000 7fd7d3f39640 1 Processor -- start 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.790+0000 7fd7d3f39640 1 -- start start 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc0d2910 0x7fd7cc0c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7cc15d330 0x7fd7cc0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0c9610 con 0x7fd7cc0d3900 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc1609d0 con 0x7fd7cc0d2910 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0c8b40 con 0x7fd7cc15d330 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42108/0 (socket says 172.21.15.124:42108) 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 -- 172.21.15.124:0/2581131471 learned_addr learned my addr 172.21.15.124:0/2581131471 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d24d6640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7cc15d330 0x7fd7cc0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d1cd5640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc0d2910 0x7fd7cc0c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 -- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7cc15d330 msgr2=0x7fd7cc0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7cc15d330 0x7fd7cc0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 -- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc0d2910 msgr2=0x7fd7cc0c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc0d2910 0x7fd7cc0c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 -- 172.21.15.124:0/2581131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c80094d0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d14d4640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00c450 tx=0x7fd7bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0052d0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc005430 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc159020 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc012870 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.791+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc0c9050 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.792+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc069400 con 0x7fd7cc0d3900 2023-12-13T23:55:05.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.793+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd7bc0129d0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.793+0000 7fd7c27fc640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 0x7fd7b006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:05.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.793+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd7bc0890c0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.793+0000 7fd7d1cd5640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 0x7fd7b006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:05.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.794+0000 7fd7d1cd5640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 0x7fd7b006d680 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8036000 tx=0x7fd7c8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:05.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.797+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd7bc0541f0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.964+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7cc0d07d0 con 0x7fd7cc0d3900 2023-12-13T23:55:05.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.965+0000 7fd7c27fc640 1 -- 172.21.15.124:0/2581131471 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12911 (secure 0 0 0) 0x7fd7bc057810 con 0x7fd7cc0d3900 2023-12-13T23:55:05.963 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:05.964 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":17,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:03.145969+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 msgr2=0x7fd7b006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 0x7fd7b006d680 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8036000 tx=0x7fd7c8008030 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 msgr2=0x7fd7cc158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00c450 tx=0x7fd7bc00c910 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 shutdown_connections 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7b006b1c0 0x7fd7b006d680 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7cc15d330 0x7fd7cc0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7cc0d3900 0x7fd7cc158ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 --2- 172.21.15.124:0/2581131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7cc0d2910 0x7fd7cc0c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 >> 172.21.15.124:0/2581131471 conn(0x7fd7cc072020 msgr2=0x7fd7cc0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 shutdown_connections 2023-12-13T23:55:05.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:05.967+0000 7fd7d3f39640 1 -- 172.21.15.124:0/2581131471 wait complete. 2023-12-13T23:55:05.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:05.976 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:06.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.080+0000 7f7bb10a6640 1 Processor -- start 2023-12-13T23:55:06.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.080+0000 7f7bb10a6640 1 -- start start 2023-12-13T23:55:06.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7bac0b4c60 0x7f7bac0d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 0x7f7bac06b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bac0b61e0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bac0d3040 con 0x7f7bac0d2bd0 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bb10a6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac0d12f0 con 0x7f7bac15d3f0 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7baa59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7bac0b4c60 0x7f7bac0d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7baa59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7bac0b4c60 0x7f7bac0d2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42116/0 (socket says 172.21.15.124:42116) 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7baa59c640 1 -- 172.21.15.124:0/3894528697 learned_addr learned my addr 172.21.15.124:0/3894528697 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7baad9d640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.081+0000 7f7bab59e640 1 --2- 172.21.15.124:0/3894528697 >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 0x7f7bac06b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 -- 172.21.15.124:0/3894528697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 msgr2=0x7f7bac0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 -- 172.21.15.124:0/3894528697 >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 msgr2=0x7f7bac06b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3894528697 >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 0x7f7bac06b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 -- 172.21.15.124:0/3894528697 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bac15d7d0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7bab59e640 1 --2- 172.21.15.124:0/3894528697 >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 0x7f7bac06b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac0d2690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0d09f0 tx=0x7f7ba002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97793f60547d1a50 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7ba9d9b640 1 -- 172.21.15.124:0/3894528697 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0030840 con 0x7f7bac0b4c60 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7ba9d9b640 1 -- 172.21.15.124:0/3894528697 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba00309a0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.082+0000 7f7ba9d9b640 1 -- 172.21.15.124:0/3894528697 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ba0030d30 con 0x7f7bac0b4c60 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3894528697 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 msgr2=0x7f7bac0d2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac0d2690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7bac0d09f0 tx=0x7f7ba002b980 comp rx=0 tx=0).stop 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3894528697 shutdown_connections 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac0d2690 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3894528697 >> v2:172.21.15.203:3300/0 conn(0x7f7bac0d2bd0 0x7f7bac06b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3894528697 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3894528697 >> 172.21.15.124:0/3894528697 conn(0x7f7bac072230 msgr2=0x7f7bac0cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3894528697 shutdown_connections 2023-12-13T23:55:06.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.083+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3894528697 wait complete. 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.084+0000 7f7bb10a6640 1 Processor -- start 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.084+0000 7f7bb10a6640 1 -- start start 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 0x7f7bac155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac1435c0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac157c20 con 0x7f7bac0d2bd0 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7bb10a6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bac157f00 con 0x7f7bac15d3f0 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7baad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7baad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42132/0 (socket says 172.21.15.124:42132) 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.085+0000 7f7baad9d640 1 -- 172.21.15.124:0/3972083160 learned_addr learned my addr 172.21.15.124:0/3972083160 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7bab59e640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 0x7f7bac155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 -- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 msgr2=0x7f7bac0ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 -- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 msgr2=0x7f7bac155c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 0x7f7bac155c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 -- 172.21.15.124:0/3972083160 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba00094d0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7bab59e640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 0x7f7bac155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7baad9d640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00efc0 tx=0x7f7b9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c009b80 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c00f040 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bac1581e0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0146a0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.086+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bac1587a0 con 0x7f7bac0b4c60 2023-12-13T23:55:06.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.087+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bac158930 con 0x7f7bac0b4c60 2023-12-13T23:55:06.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.087+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7b9c014800 con 0x7f7bac0b4c60 2023-12-13T23:55:06.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.088+0000 7f7b937fe640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 0x7f7b9806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.088+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7b9c08a200 con 0x7f7bac0b4c60 2023-12-13T23:55:06.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.088+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 0x7f7b9806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.088+0000 7f7baa59c640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 0x7f7b9806d680 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0036000 tx=0x7f7ba003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.092+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7b9c055200 con 0x7f7bac0b4c60 2023-12-13T23:55:06.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.260+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7bac069400 con 0x7f7bac0b4c60 2023-12-13T23:55:06.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.261+0000 7f7b937fe640 1 -- 172.21.15.124:0/3972083160 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b9c058820 con 0x7f7bac0b4c60 2023-12-13T23:55:06.259 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:06.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 msgr2=0x7f7b9806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 0x7f7b9806d680 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0036000 tx=0x7f7ba003b040 comp rx=0 tx=0).stop 2023-12-13T23:55:06.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 msgr2=0x7f7bac158cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c00efc0 tx=0x7f7b9c00c490 comp rx=0 tx=0).stop 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 shutdown_connections 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b9806b1c0 0x7f7b9806d680 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7bac0d2bd0 0x7f7bac155c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7bac15d3f0 0x7f7bac0ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 --2- 172.21.15.124:0/3972083160 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7bac0b4c60 0x7f7bac158cc0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 >> 172.21.15.124:0/3972083160 conn(0x7f7bac072230 msgr2=0x7f7bac0cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 shutdown_connections 2023-12-13T23:55:06.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.263+0000 7f7bb10a6640 1 -- 172.21.15.124:0/3972083160 wait complete. 2023-12-13T23:55:06.271 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:06.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.372+0000 7f2deab01640 1 Processor -- start 2023-12-13T23:55:06.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.372+0000 7f2deab01640 1 -- start start 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2deab01640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2de406ea50 0x7f2de406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2deab01640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2de40d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2deab01640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de4161d20 con 0x7f2de4160fb0 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2deab01640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de406f370 con 0x7f2de406ea50 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2deab01640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415e760 con 0x7f2de40d0ed0 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f2de4160fb0 0x7f2de4161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36906/0 (socket says 172.21.15.124:36906) 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de3fff640 1 -- 172.21.15.124:0/39176305 learned_addr learned my addr 172.21.15.124:0/39176305 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:06.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de909e640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2de40d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de889d640 1 --2- 172.21.15.124:0/39176305 >> v2:172.21.15.203:3300/0 conn(0x7f2de406ea50 0x7f2de406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277912328 0 0) 0x7f2de4161d20 con 0x7f2de4160fb0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.373+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc003520 con 0x7f2de4160fb0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457361875 0 0) 0x7f2dcc003520 con 0x7f2de4160fb0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 msgr2=0x7f2de40d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2de40d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 >> v2:172.21.15.203:3300/0 conn(0x7f2de406ea50 msgr2=0x7f2de406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 --2- 172.21.15.124:0/39176305 >> v2:172.21.15.203:3300/0 conn(0x7f2de406ea50 0x7f2de406ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de415d9a0 con 0x7f2de4160fb0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f2dd8001670 con 0x7f2de4160fb0 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 >> v1:172.21.15.124:6789/0 conn(0x7f2de4160fb0 legacy=0x7f2de4161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc003970 0x7f2dcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2dcc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc004d00 0x7f2dcc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de406f370 con 0x7f2dcc003970 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415e760 con 0x7f2dcc004d00 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc00b980 con 0x7f2de40d0ed0 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc003970 0x7f2dcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de3fff640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2dcc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de909e640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc004d00 0x7f2dcc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 -- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 msgr2=0x7f2dcc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2de40d0ed0 0x7f2dcc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 -- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc004d00 msgr2=0x7f2dcc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc004d00 0x7f2dcc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de889d640 1 -- 172.21.15.124:0/39176305 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dcc003520 con 0x7f2dcc003970 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.374+0000 7f2de909e640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc004d00 0x7f2dcc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2de889d640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc003970 0x7f2dcc003d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009ac0 tx=0x7f2dcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fab2557bbb2b4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd402c710 con 0x7f2dcc003970 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd402c870 con 0x7f2dcc003970 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2de37fe640 1 -- 172.21.15.124:0/39176305 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd402cc00 con 0x7f2dcc003970 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 -- 172.21.15.124:0/39176305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc003970 msgr2=0x7f2dcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 --2- 172.21.15.124:0/39176305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc003970 0x7f2dcc003d90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4009ac0 tx=0x7f2dcc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2de909e640 1 -- 172.21.15.124:0/39176305 reap_dead start 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 -- 172.21.15.124:0/39176305 shutdown_connections 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 -- 172.21.15.124:0/39176305 >> 172.21.15.124:0/39176305 conn(0x7f2de406a6b0 msgr2=0x7f2de406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 -- 172.21.15.124:0/39176305 shutdown_connections 2023-12-13T23:55:06.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.375+0000 7f2deab01640 1 -- 172.21.15.124:0/39176305 wait complete. 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.377+0000 7f2deab01640 1 Processor -- start 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.377+0000 7f2deab01640 1 -- start start 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.377+0000 7f2deab01640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc003970 0x7f2de415a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.377+0000 7f2deab01640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2de41567d0 0x7f2de4159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de415ad00 con 0x7f2de413f230 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de41f3b00 con 0x7f2de41567d0 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de4157ff0 con 0x7f2dcc003970 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42150/0 (socket says 172.21.15.124:42150) 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 -- 172.21.15.124:0/3734617312 learned_addr learned my addr 172.21.15.124:0/3734617312 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:06.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de3fff640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2de41567d0 0x7f2de4159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de889d640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc003970 0x7f2de415a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 -- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc003970 msgr2=0x7f2de415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc003970 0x7f2de415a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 -- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2de41567d0 msgr2=0x7f2de4159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2de41567d0 0x7f2de4159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 -- 172.21.15.124:0/3734617312 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de41582d0 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de909e640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2de40b6690 tx=0x7f2ddc0101c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc00e050 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ddc00f040 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de41585b0 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ddc010d90 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.378+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de4158b70 con 0x7f2de413f230 2023-12-13T23:55:06.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.379+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de4158d90 con 0x7f2de413f230 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.380+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2ddc015440 con 0x7f2de413f230 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.380+0000 7f2de17fa640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 0x7f2dd006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.380+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ddc0898f0 con 0x7f2de413f230 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.380+0000 7f2de889d640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 0x7f2dd006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.381+0000 7f2de889d640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 0x7f2dd006d680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036000 tx=0x7f2dd4030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:06.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.384+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2ddc054970 con 0x7f2de413f230 2023-12-13T23:55:06.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.586+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2de4069400 con 0x7f2de413f230 2023-12-13T23:55:06.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.586+0000 7f2de17fa640 1 -- 172.21.15.124:0/3734617312 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1324 (secure 0 0 0) 0x7f2ddc057f90 con 0x7f2de413f230 2023-12-13T23:55:06.584 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:06.584 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-12-13T23:55:06.585 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 2 2023-12-13T23:55:06.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 msgr2=0x7f2dd006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 0x7f2dd006d680 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2dd4036000 tx=0x7f2dd4030000 comp rx=0 tx=0).stop 2023-12-13T23:55:06.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 msgr2=0x7f2de41f3000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:06.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2de40b6690 tx=0x7f2ddc0101c0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 shutdown_connections 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b1c0 0x7f2dd006d680 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2de41567d0 0x7f2de4159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2de413f230 0x7f2de41f3000 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 --2- 172.21.15.124:0/3734617312 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc003970 0x7f2de415a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 >> 172.21.15.124:0/3734617312 conn(0x7f2de406a6b0 msgr2=0x7f2de415c520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 shutdown_connections 2023-12-13T23:55:06.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:06.588+0000 7f2deab01640 1 -- 172.21.15.124:0/3734617312 wait complete. 2023-12-13T23:55:08.597 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:08.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 Processor -- start 2023-12-13T23:55:08.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 -- start start 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f74bc06abe0 0x7f74bc15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 0x7f74bc1409d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc06e960 con 0x7f74bc1432a0 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc15dec0 con 0x7f74bc06abe0 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.714+0000 7f74c2a10640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc15e1a0 con 0x7f74bc143680 2023-12-13T23:55:08.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f74bc06abe0 0x7f74bc15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74c0fad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 0x7f74bc1409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f74bc06abe0 0x7f74bc15d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:43054/0 (socket says 172.21.15.124:43054) 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 -- 172.21.15.124:0/2345303549 learned_addr learned my addr 172.21.15.124:0/2345303549 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 -- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 msgr2=0x7f74bc1409d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 0x7f74bc1409d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 -- 172.21.15.124:0/2345303549 >> v1:172.21.15.203:6789/0 conn(0x7f74bc1432a0 legacy=0x7f74bc06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74bb7fe640 1 -- 172.21.15.124:0/2345303549 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc15e480 con 0x7f74bc06abe0 2023-12-13T23:55:08.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.715+0000 7f74c0fad640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 0x7f74bc1409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:08.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.716+0000 7f74bb7fe640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc06abe0 0x7f74bc15d8e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009870 tx=0x7f74ac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dba5314905fc0be4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:08.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.716+0000 7f74baffd640 1 -- 172.21.15.124:0/2345303549 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac030840 con 0x7f74bc06abe0 2023-12-13T23:55:08.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.716+0000 7f74baffd640 1 -- 172.21.15.124:0/2345303549 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ac0309a0 con 0x7f74bc06abe0 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.716+0000 7f74baffd640 1 -- 172.21.15.124:0/2345303549 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ac030d30 con 0x7f74bc06abe0 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 -- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc06abe0 msgr2=0x7f74bc15d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc06abe0 0x7f74bc15d8e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f74ac009870 tx=0x7f74ac02b980 comp rx=0 tx=0).stop 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 -- 172.21.15.124:0/2345303549 shutdown_connections 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc143680 0x7f74bc1409d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 --2- 172.21.15.124:0/2345303549 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc06abe0 0x7f74bc15d8e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 -- 172.21.15.124:0/2345303549 >> 172.21.15.124:0/2345303549 conn(0x7f74bc0b1de0 msgr2=0x7f74bc0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:08.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 -- 172.21.15.124:0/2345303549 shutdown_connections 2023-12-13T23:55:08.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.717+0000 7f74c2a10640 1 -- 172.21.15.124:0/2345303549 wait complete. 2023-12-13T23:55:08.719 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.718+0000 7f74c2a10640 1 Processor -- start 2023-12-13T23:55:08.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.718+0000 7f74c2a10640 1 -- start start 2023-12-13T23:55:08.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc06abe0 0x7f74bc1551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc1432a0 0x7f74bc0b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1413f0 con 0x7f74bc143680 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc140800 con 0x7f74bc1432a0 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c2a10640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc1e6d10 con 0x7f74bc06abe0 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c0fad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c0fad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42156/0 (socket says 172.21.15.124:42156) 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74c0fad640 1 -- 172.21.15.124:0/3496428890 learned_addr learned my addr 172.21.15.124:0/3496428890 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74bb7fe640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc1432a0 0x7f74bc0b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.719+0000 7f74bbfff640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc06abe0 0x7f74bc1551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 -- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc06abe0 msgr2=0x7f74bc1551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc06abe0 0x7f74bc1551c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 -- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc1432a0 msgr2=0x7f74bc0b1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc1432a0 0x7f74bc0b1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 -- 172.21.15.124:0/3496428890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b4009a70 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c0fad640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74b400efc0 tx=0x7f74b400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4009cd0 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b400f040 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac0094d0 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74b4003990 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.720+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc140d30 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.721+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc141530 con 0x7f74bc143680 2023-12-13T23:55:08.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.722+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f74b4014480 con 0x7f74bc143680 2023-12-13T23:55:08.723 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.722+0000 7f74b8ff9640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 0x7f74a806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:08.723 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.722+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f74b408a0f0 con 0x7f74bc143680 2023-12-13T23:55:08.723 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.722+0000 7f74bbfff640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 0x7f74a806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:08.723 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.723+0000 7f74bbfff640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 0x7f74a806d680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74b0000f60 tx=0x7f74b0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:08.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.726+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f74b40552b0 con 0x7f74bc143680 2023-12-13T23:55:08.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.932+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74bc160df0 con 0x7f74bc143680 2023-12-13T23:55:08.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.933+0000 7f74b8ff9640 1 -- 172.21.15.124:0/3496428890 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3703 (secure 0 0 0) 0x7f74b4018e90 con 0x7f74bc143680 2023-12-13T23:55:08.931 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:08.931 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4308,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-12-13T23:55:08.931 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 3 2023-12-13T23:55:08.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.934+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 msgr2=0x7f74a806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.934+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 0x7f74a806d680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f74b0000f60 tx=0x7f74b0009040 comp rx=0 tx=0).stop 2023-12-13T23:55:08.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 msgr2=0x7f74bc13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:08.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f74b400efc0 tx=0x7f74b400c490 comp rx=0 tx=0).stop 2023-12-13T23:55:08.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 shutdown_connections 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f74a806b1c0 0x7f74a806d680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f74bc1432a0 0x7f74bc0b1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74bc143680 0x7f74bc13c890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 --2- 172.21.15.124:0/3496428890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f74bc06abe0 0x7f74bc1551c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 >> 172.21.15.124:0/3496428890 conn(0x7f74bc0b1de0 msgr2=0x7f74bc13f820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 shutdown_connections 2023-12-13T23:55:08.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:08.935+0000 7f74c2a10640 1 -- 172.21.15.124:0/3496428890 wait complete. 2023-12-13T23:55:08.946 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.047+0000 7fd1d8dd7640 1 Processor -- start 2023-12-13T23:55:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.047+0000 7fd1d8dd7640 1 -- start start 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d8dd7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd1d41448f0 0x7fd1d415ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d8dd7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd1d415b5d0 0x7fd1d415b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d8dd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d415bef0 con 0x7fd1d406a4d0 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d8dd7640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d40714c0 con 0x7fd1d415b5d0 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d8dd7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1d415f100 con 0x7fd1d41448f0 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd1d415b5d0 0x7fd1d415b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd1d415b5d0 0x7fd1d415b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42166/0 (socket says 172.21.15.124:42166) 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 -- 172.21.15.124:0/824595847 learned_addr learned my addr 172.21.15.124:0/824595847 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d259c640 1 --2- 172.21.15.124:0/824595847 >> v2:172.21.15.203:3300/0 conn(0x7fd1d41448f0 0x7fd1d415ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 -- 172.21.15.124:0/824595847 >> v2:172.21.15.203:3300/0 conn(0x7fd1d41448f0 msgr2=0x7fd1d415ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 --2- 172.21.15.124:0/824595847 >> v2:172.21.15.203:3300/0 conn(0x7fd1d41448f0 0x7fd1d415ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 -- 172.21.15.124:0/824595847 >> v1:172.21.15.203:6789/0 conn(0x7fd1d406a4d0 legacy=0x7fd1d4071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.048+0000 7fd1d2d9d640 1 -- 172.21.15.124:0/824595847 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d415f3a0 con 0x7fd1d415b5d0 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.049+0000 7fd1d2d9d640 1 --2- 172.21.15.124:0/824595847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d415b5d0 0x7fd1d415b9b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40b55f0 tx=0x7fd1cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b84fdf4c26df80 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.049+0000 7fd1d159a640 1 -- 172.21.15.124:0/824595847 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc031870 con 0x7fd1d415b5d0 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.049+0000 7fd1d159a640 1 -- 172.21.15.124:0/824595847 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1cc0319d0 con 0x7fd1d415b5d0 2023-12-13T23:55:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.049+0000 7fd1d159a640 1 -- 172.21.15.124:0/824595847 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc031d60 con 0x7fd1d415b5d0 2023-12-13T23:55:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/824595847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d415b5d0 msgr2=0x7fd1d415b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/824595847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d415b5d0 0x7fd1d415b9b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1d40b55f0 tx=0x7fd1cc02b910 comp rx=0 tx=0).stop 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/824595847 shutdown_connections 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/824595847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d415b5d0 0x7fd1d415b9b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/824595847 >> v2:172.21.15.203:3300/0 conn(0x7fd1d41448f0 0x7fd1d415ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/824595847 >> 172.21.15.124:0/824595847 conn(0x7fd1d40b1de0 msgr2=0x7fd1d40b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/824595847 shutdown_connections 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.050+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/824595847 wait complete. 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.051+0000 7fd1d8dd7640 1 Processor -- start 2023-12-13T23:55:09.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.051+0000 7fd1d8dd7640 1 -- start start 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 0x7fd1d413b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 0x7fd1d40cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d413a3c0 con 0x7fd1d41448f0 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d413a6a0 con 0x7fd1d415b5d0 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d8dd7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d413a980 con 0x7fd1d406a4d0 2023-12-13T23:55:09.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42178/0 (socket says 172.21.15.124:42178) 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 -- 172.21.15.124:0/2928918742 learned_addr learned my addr 172.21.15.124:0/2928918742 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d259c640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 0x7fd1d413b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d2d9d640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 0x7fd1d40cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 -- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 msgr2=0x7fd1d413b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 0x7fd1d413b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 -- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 msgr2=0x7fd1d40cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 0x7fd1d40cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 -- 172.21.15.124:0/2928918742 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0094d0 con 0x7fd1d41448f0 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d259c640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 0x7fd1d413b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d2d9d640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 0x7fd1d40cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.052+0000 7fd1d1d9b640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c430 tx=0x7fd1c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.053+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011790 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.053+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1c80118f0 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.053+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d413ac60 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.053+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1c8011ce0 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.053+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d413a060 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.054+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1d413b180 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.054+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd1c801c460 con 0x7fd1d41448f0 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.054+0000 7fd1c2ffd640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 0x7fd1b406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.055+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd1c8089690 con 0x7fd1d41448f0 2023-12-13T23:55:09.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.055+0000 7fd1d259c640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 0x7fd1b406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.055+0000 7fd1d259c640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 0x7fd1b406d680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1d413f1a0 tx=0x7fd1c400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.059+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd1d413b180 con 0x7fd1d41448f0 2023-12-13T23:55:09.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.230+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1d413b180 con 0x7fd1d41448f0 2023-12-13T23:55:09.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.231+0000 7fd1c2ffd640 1 -- 172.21.15.124:0/2928918742 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12911 (secure 0 0 0) 0x7fd1d413b180 con 0x7fd1d41448f0 2023-12-13T23:55:09.229 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:09.229 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":17,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:03.145969+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 msgr2=0x7fd1b406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 0x7fd1b406d680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd1d413f1a0 tx=0x7fd1c400b040 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 msgr2=0x7fd1d415d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1c800c430 tx=0x7fd1c800c8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 shutdown_connections 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd1b406b1c0 0x7fd1b406d680 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd1d415b5d0 0x7fd1d40cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd1d41448f0 0x7fd1d415d400 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 --2- 172.21.15.124:0/2928918742 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd1d406a4d0 0x7fd1d413b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 >> 172.21.15.124:0/2928918742 conn(0x7fd1d40b1de0 msgr2=0x7fd1d4142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 shutdown_connections 2023-12-13T23:55:09.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.232+0000 7fd1d8dd7640 1 -- 172.21.15.124:0/2928918742 wait complete. 2023-12-13T23:55:09.241 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:09.242 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:09.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.345+0000 7f61aa92c640 1 Processor -- start 2023-12-13T23:55:09.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 -- start start 2023-12-13T23:55:09.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f61a415b130 0x7f61a40d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a40cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f61a40cf150 0x7f61a406ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a415fd20 con 0x7f61a415b130 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a40d0730 con 0x7f61a40cf150 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61aa92c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40cf5c0 con 0x7f61a415f760 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61a3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f61a415b130 0x7f61a40d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61a3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f61a415b130 0x7f61a40d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42184/0 (socket says 172.21.15.124:42184) 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.346+0000 7f61a3fff640 1 -- 172.21.15.124:0/1300323820 learned_addr learned my addr 172.21.15.124:0/1300323820 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a37fe640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a40cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a8ec9640 1 --2- 172.21.15.124:0/1300323820 >> v2:172.21.15.203:3300/0 conn(0x7f61a40cf150 0x7f61a406ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 -- 172.21.15.124:0/1300323820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 msgr2=0x7f61a40cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a40cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 -- 172.21.15.124:0/1300323820 >> v2:172.21.15.203:3300/0 conn(0x7f61a40cf150 msgr2=0x7f61a406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 --2- 172.21.15.124:0/1300323820 >> v2:172.21.15.203:3300/0 conn(0x7f61a40cf150 0x7f61a406ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 -- 172.21.15.124:0/1300323820 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40b3f10 con 0x7f61a415b130 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.347+0000 7f61a3fff640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a415b130 0x7f61a40d01f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61a40cce80 tx=0x7f619402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fad0680b257fc93c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61a2ffd640 1 -- 172.21.15.124:0/1300323820 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031640 con 0x7f61a415b130 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61a2ffd640 1 -- 172.21.15.124:0/1300323820 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61940317a0 con 0x7f61a415b130 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61a2ffd640 1 -- 172.21.15.124:0/1300323820 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031b30 con 0x7f61a415b130 2023-12-13T23:55:09.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 -- 172.21.15.124:0/1300323820 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a415b130 msgr2=0x7f61a40d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a415b130 0x7f61a40d01f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61a40cce80 tx=0x7f619402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 -- 172.21.15.124:0/1300323820 shutdown_connections 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 --2- 172.21.15.124:0/1300323820 >> v2:172.21.15.203:3300/0 conn(0x7f61a40cf150 0x7f61a406ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a40cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 --2- 172.21.15.124:0/1300323820 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a415b130 0x7f61a40d01f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 -- 172.21.15.124:0/1300323820 >> 172.21.15.124:0/1300323820 conn(0x7f61a40b2c10 msgr2=0x7f61a40b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 -- 172.21.15.124:0/1300323820 shutdown_connections 2023-12-13T23:55:09.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.348+0000 7f61aa92c640 1 -- 172.21.15.124:0/1300323820 wait complete. 2023-12-13T23:55:09.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.349+0000 7f61aa92c640 1 Processor -- start 2023-12-13T23:55:09.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.349+0000 7f61aa92c640 1 -- start start 2023-12-13T23:55:09.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 0x7f61a40c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a41586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4158c00 con 0x7f61a40cf150 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40c9860 con 0x7f61a415b130 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61aa92c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a40c9b40 con 0x7f61a415f760 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42196/0 (socket says 172.21.15.124:42196) 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a3fff640 1 -- 172.21.15.124:0/2418142556 learned_addr learned my addr 172.21.15.124:0/2418142556 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a8ec9640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a41586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a37fe640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 0x7f61a40c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.350+0000 7f61a3fff640 1 -- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 msgr2=0x7f61a41586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a3fff640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a41586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a3fff640 1 -- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 msgr2=0x7f61a40c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a3fff640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 0x7f61a40c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a3fff640 1 -- 172.21.15.124:0/2418142556 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61940094d0 con 0x7f61a40cf150 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a8ec9640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a41586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a37fe640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 0x7f61a40c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a3fff640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f619402b930 tx=0x7f61940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619403ac60 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619403ade0 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194044630 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a40c9e20 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.351+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a40ca3e0 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.352+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a4067fc0 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.352+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f619402f050 con 0x7f61a40cf150 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.353+0000 7f61a0ff9640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 0x7f619006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.353+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f61940ab530 con 0x7f61a40cf150 2023-12-13T23:55:09.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.353+0000 7f61a37fe640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 0x7f619006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.353+0000 7f61a37fe640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 0x7f619006d680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6198006fd0 tx=0x7f6198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.357+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6194076640 con 0x7f61a40cf150 2023-12-13T23:55:09.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.523+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f61a4067420 con 0x7f61a40cf150 2023-12-13T23:55:09.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.523+0000 7f61a0ff9640 1 -- 172.21.15.124:0/2418142556 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f61940350a0 con 0x7f61a40cf150 2023-12-13T23:55:09.521 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 msgr2=0x7f619006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 0x7f619006d680 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6198006fd0 tx=0x7f6198008040 comp rx=0 tx=0).stop 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 msgr2=0x7f61a40b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f619402b930 tx=0x7f61940046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 shutdown_connections 2023-12-13T23:55:09.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f619006b1c0 0x7f619006d680 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f61a415b130 0x7f61a40c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f61a415f760 0x7f61a41586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 --2- 172.21.15.124:0/2418142556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f61a40cf150 0x7f61a40b47d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 >> 172.21.15.124:0/2418142556 conn(0x7f61a40b2c10 msgr2=0x7f61a40d1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 shutdown_connections 2023-12-13T23:55:09.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.525+0000 7f61aa92c640 1 -- 172.21.15.124:0/2418142556 wait complete. 2023-12-13T23:55:09.534 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-13T23:55:09.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.636+0000 7fbbd81a9640 1 Processor -- start 2023-12-13T23:55:09.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.636+0000 7fbbd81a9640 1 -- start start 2023-12-13T23:55:09.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd81a9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbbd006f210 0x7fbbd0140800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd81a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd015ba40 con 0x7fbbd0140d40 2023-12-13T23:55:09.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd81a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd0141190 con 0x7fbbd015c2f0 2023-12-13T23:55:09.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd81a9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd015c6d0 con 0x7fbbd006f210 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd5744640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbbd006f210 0x7fbbd0140800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd6746640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fbbd0140d40 0x7fbbd013f330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36910/0 (socket says 172.21.15.124:36910) 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd6746640 1 -- 172.21.15.124:0/3266781828 learned_addr learned my addr 172.21.15.124:0/3266781828 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd5744640 1 -- 172.21.15.124:0/3266781828 >> v1:172.21.15.203:6789/0 conn(0x7fbbd015c2f0 legacy=0x7fbbd006eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd5744640 1 -- 172.21.15.124:0/3266781828 >> v1:172.21.15.124:6789/0 conn(0x7fbbd0140d40 legacy=0x7fbbd013f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd4f43640 1 -- 172.21.15.124:0/3266781828 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778101966 0 0) 0x7fbbd015ba40 con 0x7fbbd0140d40 2023-12-13T23:55:09.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.637+0000 7fbbd5744640 1 -- 172.21.15.124:0/3266781828 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd01416e0 con 0x7fbbd006f210 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.638+0000 7fbbd5744640 1 --2- 172.21.15.124:0/3266781828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0140800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbd01418d0 tx=0x7fbbc002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb34c524fe26af18 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.638+0000 7fbbd4f43640 1 -- 172.21.15.124:0/3266781828 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0031830 con 0x7fbbd006f210 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.638+0000 7fbbd4f43640 1 -- 172.21.15.124:0/3266781828 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc0031990 con 0x7fbbd006f210 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.638+0000 7fbbd4f43640 1 -- 172.21.15.124:0/3266781828 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0031d20 con 0x7fbbd006f210 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.638+0000 7fbbd81a9640 1 -- 172.21.15.124:0/3266781828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 msgr2=0x7fbbd0140800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/3266781828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0140800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbd01418d0 tx=0x7fbbc002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 -- 172.21.15.124:0/3266781828 shutdown_connections 2023-12-13T23:55:09.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/3266781828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0140800 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 -- 172.21.15.124:0/3266781828 >> 172.21.15.124:0/3266781828 conn(0x7fbbd006a6b0 msgr2=0x7fbbd006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 -- 172.21.15.124:0/3266781828 shutdown_connections 2023-12-13T23:55:09.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.639+0000 7fbbd81a9640 1 -- 172.21.15.124:0/3266781828 wait complete. 2023-12-13T23:55:09.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 Processor -- start 2023-12-13T23:55:09.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 -- start start 2023-12-13T23:55:09.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbd0140d40 0x7fbbd00b54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 0x7fbbd0151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01f4f70 con 0x7fbbd006f210 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01f4360 con 0x7fbbd0140d40 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.640+0000 7fbbd81a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd01f4640 con 0x7fbbd015c2f0 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42212/0 (socket says 172.21.15.124:42212) 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 -- 172.21.15.124:0/1378635198 learned_addr learned my addr 172.21.15.124:0/1378635198 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5744640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbd0140d40 0x7fbbd00b54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd6746640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 0x7fbbd0151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 -- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 msgr2=0x7fbbd0151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 0x7fbbd0151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 -- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbd0140d40 msgr2=0x7fbbd00b54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbd0140d40 0x7fbbd00b54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 -- 172.21.15.124:0/1378635198 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc0009400 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd6746640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 0x7fbbd0151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd5f45640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00b3f0 tx=0x7fbbcc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc011830 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc011990 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd01f4920 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc011d40 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.641+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd0153c80 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.642+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd0069400 con 0x7fbbd006f210 2023-12-13T23:55:09.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.642+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbbcc01a400 con 0x7fbbd006f210 2023-12-13T23:55:09.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.643+0000 7fbbc67fc640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 0x7fbbb406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.643+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbbcc0893f0 con 0x7fbbd006f210 2023-12-13T23:55:09.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.643+0000 7fbbd5744640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 0x7fbbb406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.644+0000 7fbbd5744640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 0x7fbbb406d680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0036000 tx=0x7fbbc003d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.647+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbbcc054500 con 0x7fbbd006f210 2023-12-13T23:55:09.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.816+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fbbd0070d30 con 0x7fbbd006f210 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.816+0000 7fbbc67fc640 1 -- 172.21.15.124:0/1378635198 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7fbbcc057b20 con 0x7fbbd006f210 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout:[ 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout: "poolnum": 1, 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout: "poolname": ".mgr" 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:55:09.815 INFO:teuthology.orchestra.run.smithi124.stdout:] 2023-12-13T23:55:09.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 msgr2=0x7fbbb406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 0x7fbbb406d680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0036000 tx=0x7fbbc003d040 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 msgr2=0x7fbbd0155cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00b3f0 tx=0x7fbbcc00b8b0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 shutdown_connections 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbbb406b1c0 0x7fbbb406d680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbbd0140d40 0x7fbbd00b54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbbd015c2f0 0x7fbbd0151140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 --2- 172.21.15.124:0/1378635198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbbd006f210 0x7fbbd0155cb0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 >> 172.21.15.124:0/1378635198 conn(0x7fbbd006a6b0 msgr2=0x7fbbd006eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 shutdown_connections 2023-12-13T23:55:09.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.818+0000 7fbbd81a9640 1 -- 172.21.15.124:0/1378635198 wait complete. 2023-12-13T23:55:09.827 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-13T23:55:09.827 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-13T23:55:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.932+0000 7fc0ec097640 1 Processor -- start 2023-12-13T23:55:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 -- start start 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc0e40d2910 0x7fc0e40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 0x7fc0e40d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40b2b10 con 0x7fc0e40d2910 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e40d1590 con 0x7fc0e40d2d90 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0ec097640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40d1870 con 0x7fc0e415d330 2023-12-13T23:55:09.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.933+0000 7fc0e9632640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc0e40d2910 0x7fc0e40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc0e40d2910 0x7fc0e40d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42214/0 (socket says 172.21.15.124:42214) 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 -- 172.21.15.124:0/3394193930 learned_addr learned my addr 172.21.15.124:0/3394193930 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9e33640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0ea634640 1 --2- 172.21.15.124:0/3394193930 >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 0x7fc0e40d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 -- 172.21.15.124:0/3394193930 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 msgr2=0x7fc0e40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e40b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 -- 172.21.15.124:0/3394193930 >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 msgr2=0x7fc0e40d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3394193930 >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 0x7fc0e40d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 -- 172.21.15.124:0/3394193930 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e40d1b50 con 0x7fc0e40d2910 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9e33640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e40b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:09.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0ea634640 1 --2- 172.21.15.124:0/3394193930 >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 0x7fc0e40d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:09.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.934+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40d31f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40d1540 tx=0x7fc0d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97219755e063e0f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0e8e31640 1 -- 172.21.15.124:0/3394193930 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d40317e0 con 0x7fc0e40d2910 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0e8e31640 1 -- 172.21.15.124:0/3394193930 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d4031940 con 0x7fc0e40d2910 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0e8e31640 1 -- 172.21.15.124:0/3394193930 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4031cd0 con 0x7fc0e40d2910 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 -- 172.21.15.124:0/3394193930 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 msgr2=0x7fc0e40d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40d31f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc0e40d1540 tx=0x7fc0d402b880 comp rx=0 tx=0).stop 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 -- 172.21.15.124:0/3394193930 shutdown_connections 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40d31f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3394193930 >> v2:172.21.15.203:3300/0 conn(0x7fc0e40d2d90 0x7fc0e40d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3394193930 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e40b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 -- 172.21.15.124:0/3394193930 >> 172.21.15.124:0/3394193930 conn(0x7fc0e4072020 msgr2=0x7fc0e4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 -- 172.21.15.124:0/3394193930 shutdown_connections 2023-12-13T23:55:09.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.935+0000 7fc0ec097640 1 -- 172.21.15.124:0/3394193930 wait complete. 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.936+0000 7fc0ec097640 1 Processor -- start 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.936+0000 7fc0ec097640 1 -- start start 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 0x7fc0e40c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e415f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40c7870 con 0x7fc0e40d2910 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40c7b50 con 0x7fc0e40d2d90 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ec097640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e40c7e30 con 0x7fc0e415d330 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42216/0 (socket says 172.21.15.124:42216) 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 -- 172.21.15.124:0/3776172863 learned_addr learned my addr 172.21.15.124:0/3776172863 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0ea634640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e415f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 0x7fc0e40c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 -- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 msgr2=0x7fc0e415f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e415f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 -- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 msgr2=0x7fc0e40c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.937+0000 7fc0e9e33640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 0x7fc0e40c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0e9e33640 1 -- 172.21.15.124:0/3776172863 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d40094d0 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 0x7fc0e40c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0e9e33640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000e7e0 tx=0x7fc0e000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004030 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0004190 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e40c8110 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004520 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.938+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e40c7580 con 0x7fc0e40d2910 2023-12-13T23:55:09.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.939+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e4069400 con 0x7fc0e40d2910 2023-12-13T23:55:09.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.939+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc0e000c6d0 con 0x7fc0e40d2910 2023-12-13T23:55:09.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.939+0000 7fc0da7fc640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 0x7fc0c806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:09.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.940+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc0e0089370 con 0x7fc0e40d2910 2023-12-13T23:55:09.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.940+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 0x7fc0c806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:09.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.940+0000 7fc0e9632640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 0x7fc0c806d6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4036000 tx=0x7fc0d403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:09.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:09.944+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc0e00543f0 con 0x7fc0e40d2910 2023-12-13T23:55:10.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.110+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc0e40c79b0 con 0x7fc0e40d2910 2023-12-13T23:55:10.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.739+0000 7fc0da7fc640 1 -- 172.21.15.124:0/3776172863 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc0e40c79b0 con 0x7fc0e40d2910 2023-12-13T23:55:10.737 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' created 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 msgr2=0x7fc0c806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 0x7fc0c806d6b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4036000 tx=0x7fc0d403d040 comp rx=0 tx=0).stop 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 msgr2=0x7fc0e40c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc0e000e7e0 tx=0x7fc0e000eca0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 shutdown_connections 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc0c806b1f0 0x7fc0c806d6b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0e40d2d90 0x7fc0e40c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0e415d330 0x7fc0e415f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 --2- 172.21.15.124:0/3776172863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0e40d2910 0x7fc0e40c7040 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 >> 172.21.15.124:0/3776172863 conn(0x7fc0e4072020 msgr2=0x7fc0e40b63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:10.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 shutdown_connections 2023-12-13T23:55:10.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.742+0000 7fc0ec097640 1 -- 172.21.15.124:0/3776172863 wait complete. 2023-12-13T23:55:10.755 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-13T23:55:10.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.865+0000 7f0c1983d640 1 Processor -- start 2023-12-13T23:55:10.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.865+0000 7f0c1983d640 1 -- start start 2023-12-13T23:55:10.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c1983d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 0x7f0c140b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c1983d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c1983d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c140b7000 con 0x7f0c14099fb0 2023-12-13T23:55:10.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c1983d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c140b29f0 con 0x7f0c140cd4a0 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c1983d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140cd920 con 0x7f0c140b6c20 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c127fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c12ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 0x7f0c140b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c127fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:43842/0 (socket says 172.21.15.124:43842) 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c127fc640 1 -- 172.21.15.124:0/1338160308 learned_addr learned my addr 172.21.15.124:0/1338160308 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.866+0000 7f0c137fe640 1 --1- 172.21.15.124:0/1338160308 >> v1:172.21.15.203:6789/0 conn(0x7f0c14099fb0 0x7f0c1415f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:59218/0 (socket says 172.21.15.124:59218) 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c12ffd640 1 -- 172.21.15.124:0/1338160308 >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 msgr2=0x7f0c14099a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c12ffd640 1 --2- 172.21.15.124:0/1338160308 >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c12ffd640 1 -- 172.21.15.124:0/1338160308 >> v1:172.21.15.203:6789/0 conn(0x7f0c14099fb0 legacy=0x7f0c1415f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c12ffd640 1 -- 172.21.15.124:0/1338160308 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c140d1db0 con 0x7f0c140b6c20 2023-12-13T23:55:10.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c127fc640 1 --2- 172.21.15.124:0/1338160308 >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.867+0000 7f0c12ffd640 1 --2- 172.21.15.124:0/1338160308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 0x7f0c140b1b90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c0402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=433502fb8ddadabc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c11ffb640 1 -- 172.21.15.124:0/1338160308 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0402cc50 con 0x7f0c140b6c20 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c11ffb640 1 -- 172.21.15.124:0/1338160308 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c0402cdb0 con 0x7f0c140b6c20 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c11ffb640 1 -- 172.21.15.124:0/1338160308 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04034660 con 0x7f0c140b6c20 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 -- 172.21.15.124:0/1338160308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 msgr2=0x7f0c140b1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 --2- 172.21.15.124:0/1338160308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 0x7f0c140b1b90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c0402bea0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 -- 172.21.15.124:0/1338160308 shutdown_connections 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 --2- 172.21.15.124:0/1338160308 >> v2:172.21.15.203:3300/0 conn(0x7f0c140cd4a0 0x7f0c14099a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 --2- 172.21.15.124:0/1338160308 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140b6c20 0x7f0c140b1b90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 -- 172.21.15.124:0/1338160308 >> 172.21.15.124:0/1338160308 conn(0x7f0c140d1000 msgr2=0x7f0c140d1410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:10.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 -- 172.21.15.124:0/1338160308 shutdown_connections 2023-12-13T23:55:10.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.868+0000 7f0c1983d640 1 -- 172.21.15.124:0/1338160308 wait complete. 2023-12-13T23:55:10.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.869+0000 7f0c1983d640 1 Processor -- start 2023-12-13T23:55:10.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 -- start start 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c140b6c20 0x7f0c140ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140cd4a0 0x7f0c140af560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140c7470 con 0x7f0c14099fb0 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140c6880 con 0x7f0c140b6c20 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c1983d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c140c6ad0 con 0x7f0c140cd4a0 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c12ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c12ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48588/0 (socket says 172.21.15.124:48588) 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.870+0000 7f0c12ffd640 1 -- 172.21.15.124:0/2499100289 learned_addr learned my addr 172.21.15.124:0/2499100289 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c127fc640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c140b6c20 0x7f0c140ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c137fe640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140cd4a0 0x7f0c140af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 -- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140cd4a0 msgr2=0x7f0c140af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140cd4a0 0x7f0c140af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 -- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c140b6c20 msgr2=0x7f0c140ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c140b6c20 0x7f0c140ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 -- 172.21.15.124:0/2499100289 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c080094d0 con 0x7f0c14099fb0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c12ffd640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04040650 con 0x7f0c14099fb0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c040407b0 con 0x7f0c14099fb0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c04009a70 con 0x7f0c14099fb0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04040b60 con 0x7f0c14099fb0 2023-12-13T23:55:10.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.871+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c140c7090 con 0x7f0c14099fb0 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.872+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c140c75b0 con 0x7f0c14099fb0 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.872+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0c04040cc0 con 0x7f0c14099fb0 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.873+0000 7f0bfbfff640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 0x7f0c0006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.873+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f0c040ac4f0 con 0x7f0c14099fb0 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.873+0000 7f0c127fc640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 0x7f0c0006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:10.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.873+0000 7f0c127fc640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 0x7f0c0006d680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c140cea90 tx=0x7f0c08009240 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:10.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:10.877+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0c140c75b0 con 0x7f0c14099fb0 2023-12-13T23:55:11.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.046+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0c140c75b0 con 0x7f0c14099fb0 2023-12-13T23:55:11.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:55:11.681 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:55:11.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.745+0000 7f0bfbfff640 1 -- 172.21.15.124:0/2499100289 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0c140c75b0 con 0x7f0c14099fb0 2023-12-13T23:55:11.743 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' created 2023-12-13T23:55:11.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 msgr2=0x7f0c0006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 0x7f0c0006d680 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c140cea90 tx=0x7f0c08009240 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 msgr2=0x7f0c140ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c04004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 shutdown_connections 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c0006b1c0 0x7f0c0006d680 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c140b6c20 0x7f0c140ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c140cd4a0 0x7f0c140af560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 --2- 172.21.15.124:0/2499100289 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c14099fb0 0x7f0c140ae300 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009d90 tx=0x7f0c04004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 >> 172.21.15.124:0/2499100289 conn(0x7f0c140d1000 msgr2=0x7f0c1406f990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:11.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.748+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 shutdown_connections 2023-12-13T23:55:11.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.749+0000 7f0c1983d640 1 -- 172.21.15.124:0/2499100289 wait complete. 2023-12-13T23:55:11.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-13T23:55:11.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.910+0000 7f989d148640 1 Processor -- start 2023-12-13T23:55:11.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.910+0000 7f989d148640 1 -- start start 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989d148640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f98981618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989d148640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f98981409d0 0x7f989806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989d148640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98981406f0 con 0x7f9898140310 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989d148640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9898161e20 con 0x7f98981409d0 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989d148640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9898143960 con 0x7f989806abe0 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f9896d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9898140310 0x7f989806a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46638/0 (socket says 172.21.15.124:46638) 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f9896d9d640 1 -- 172.21.15.124:0/1451023797 learned_addr learned my addr 172.21.15.124:0/1451023797 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989659c640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f98981618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f989759e640 1 --2- 172.21.15.124:0/1451023797 >> v2:172.21.15.203:3300/0 conn(0x7f98981409d0 0x7f989806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 832001478 0 0) 0x7f98981406f0 con 0x7f9898140310 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.911+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9880003520 con 0x7f9898140310 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3118721670 0 0) 0x7f9880003520 con 0x7f9898140310 2023-12-13T23:55:11.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 msgr2=0x7f98981618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f98981618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 >> v2:172.21.15.203:3300/0 conn(0x7f98981409d0 msgr2=0x7f989806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 --2- 172.21.15.124:0/1451023797 >> v2:172.21.15.203:3300/0 conn(0x7f98981409d0 0x7f989806efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9898142de0 con 0x7f9898140310 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f98880019f0 con 0x7f9898140310 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 >> v1:172.21.15.124:6789/0 conn(0x7f9898140310 legacy=0x7f989806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9880003970 0x7f9880003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f98800047c0 0x7f9880004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f9880005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9898161e20 con 0x7f98800047c0 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9898143960 con 0x7f9880003970 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988000b980 con 0x7f989806abe0 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f98800047c0 0x7f9880004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f9896d9d640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9880003970 0x7f9880003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989759e640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f9880005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 -- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 msgr2=0x7f9880005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989806abe0 0x7f9880005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 -- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9880003970 msgr2=0x7f9880003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9880003970 0x7f9880003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.912+0000 7f989659c640 1 -- 172.21.15.124:0/1451023797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880003520 con 0x7f98800047c0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989659c640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f98800047c0 0x7f9880004c40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f988c009af0 tx=0x7f98800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2207f741ed654f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c02c6b0 con 0x7f98800047c0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c02c810 con 0x7f98800047c0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f9895d9b640 1 -- 172.21.15.124:0/1451023797 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c02cba0 con 0x7f98800047c0 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989d148640 1 -- 172.21.15.124:0/1451023797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f98800047c0 msgr2=0x7f9880004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989d148640 1 --2- 172.21.15.124:0/1451023797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f98800047c0 0x7f9880004c40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f988c009af0 tx=0x7f98800057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989759e640 1 -- 172.21.15.124:0/1451023797 reap_dead start 2023-12-13T23:55:11.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989d148640 1 -- 172.21.15.124:0/1451023797 shutdown_connections 2023-12-13T23:55:11.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989d148640 1 -- 172.21.15.124:0/1451023797 >> 172.21.15.124:0/1451023797 conn(0x7f98980b1e50 msgr2=0x7f98980b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:11.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.913+0000 7f989d148640 1 -- 172.21.15.124:0/1451023797 shutdown_connections 2023-12-13T23:55:11.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.914+0000 7f989d148640 1 -- 172.21.15.124:0/1451023797 wait complete. 2023-12-13T23:55:11.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.915+0000 7f989d148640 1 Processor -- start 2023-12-13T23:55:11.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.915+0000 7f989d148640 1 -- start start 2023-12-13T23:55:11.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 0x7f989815dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989812f210 0x7f989806f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9898146bd0 con 0x7f989812ed90 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9898143c60 con 0x7f98800047c0 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9898143f40 con 0x7f989812f210 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48598/0 (socket says 172.21.15.124:48598) 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 -- 172.21.15.124:0/811812217 learned_addr learned my addr 172.21.15.124:0/811812217 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989759e640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989812f210 0x7f989806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f9896d9d640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 0x7f989815dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 -- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989812f210 msgr2=0x7f989806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989812f210 0x7f989806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 -- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 msgr2=0x7f989815dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 0x7f989815dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:11.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 -- 172.21.15.124:0/811812217 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989812e590 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f9896d9d640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 0x7f989815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989659c640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f988c037040 tx=0x7f988c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c003810 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f988c003970 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989812e870 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988c003ce0 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.916+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9898144400 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.917+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989812ea00 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.918+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f988c047450 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.918+0000 7f987f7fe640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 0x7f988406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.918+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f988c0abae0 con 0x7f989812ed90 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.918+0000 7f9896d9d640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 0x7f988406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:11.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.919+0000 7f9896d9d640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 0x7f988406d6b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98880013d0 tx=0x7f9888006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:11.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:11.923+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f988c076ad0 con 0x7f989812ed90 2023-12-13T23:55:12.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.123+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9898144080 con 0x7f989812ed90 2023-12-13T23:55:12.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.748+0000 7f987f7fe640 1 -- 172.21.15.124:0/811812217 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9898144080 con 0x7f989812ed90 2023-12-13T23:55:12.747 INFO:teuthology.orchestra.run.smithi124.stderr:new fs with metadata pool 2 and data pool 3 2023-12-13T23:55:12.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 msgr2=0x7f988406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 0x7f988406d6b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98880013d0 tx=0x7f9888006040 comp rx=0 tx=0).stop 2023-12-13T23:55:12.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 msgr2=0x7f989806f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f988c037040 tx=0x7f988c004730 comp rx=0 tx=0).stop 2023-12-13T23:55:12.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 shutdown_connections 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f988406b1f0 0x7f988406d6b0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f98880013d0 tx=0x7f9888006040 comp rx=0 tx=0).stop 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f989812f210 0x7f989806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f989812ed90 0x7f989806f300 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f988c037040 tx=0x7f988c004730 comp rx=0 tx=0).stop 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 --2- 172.21.15.124:0/811812217 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f98800047c0 0x7f989815dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.752+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 >> 172.21.15.124:0/811812217 conn(0x7f98980b1e50 msgr2=0x7f98980733b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.753+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 shutdown_connections 2023-12-13T23:55:12.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.753+0000 7f989d148640 1 -- 172.21.15.124:0/811812217 wait complete. 2023-12-13T23:55:12.771 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:12.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.875+0000 7f9cf8fc4640 1 Processor -- start 2023-12-13T23:55:12.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.875+0000 7f9cf8fc4640 1 -- start start 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf8fc4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9cf415d330 0x7f9cf40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf8fc4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cf40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf8fc4640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf40b2b10 con 0x7f9cf40d2910 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf8fc4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf40d1590 con 0x7f9cf415d330 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf8fc4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40d1870 con 0x7f9cf40d3900 2023-12-13T23:55:12.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf1d9b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9cf40d2910 0x7f9cf40d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46654/0 (socket says 172.21.15.124:46654) 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.876+0000 7f9cf1d9b640 1 -- 172.21.15.124:0/2776620134 learned_addr learned my addr 172.21.15.124:0/2776620134 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505671456 0 0) 0x7f9cf40b2b10 con 0x7f9cf40d2910 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf259c640 1 --2- 172.21.15.124:0/2776620134 >> v2:172.21.15.203:3300/0 conn(0x7f9cf415d330 0x7f9cf40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf2d9d640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cf40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc003520 con 0x7f9cf40d2910 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 462921892 0 0) 0x7f9cdc003520 con 0x7f9cf40d2910 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 msgr2=0x7f9cf40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cf40b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 >> v2:172.21.15.203:3300/0 conn(0x7f9cf415d330 msgr2=0x7f9cf40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 --2- 172.21.15.124:0/2776620134 >> v2:172.21.15.203:3300/0 conn(0x7f9cf415d330 0x7f9cf40b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf40d1b50 con 0x7f9cf40d2910 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f9ce8001670 con 0x7f9cf40d2910 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 >> v1:172.21.15.124:6789/0 conn(0x7f9cf40d2910 legacy=0x7f9cf40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cdc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cdc003ef0 0x7f9cdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40d1590 con 0x7f9cdc003ef0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40d1870 con 0x7f9cdc004d00 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00b9d0 con 0x7f9cf40d3900 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cdc003ef0 0x7f9cdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf259c640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cdc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf2d9d640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 -- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 msgr2=0x7f9cdc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cf40d3900 0x7f9cdc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 -- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc004d00 msgr2=0x7f9cdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.877+0000 7f9cf1d9b640 1 -- 172.21.15.124:0/2776620134 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc003520 con 0x7f9cdc003ef0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.878+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cdc003ef0 0x7f9cdc0047c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c2c0 tx=0x7f9cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1457f554e3f7c017 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:12.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.878+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8005ad0 con 0x7f9cdc003ef0 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.878+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce8005c30 con 0x7f9cdc003ef0 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.878+0000 7f9cf159a640 1 -- 172.21.15.124:0/2776620134 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce802c8b0 con 0x7f9cdc003ef0 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/2776620134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cdc003ef0 msgr2=0x7f9cdc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/2776620134 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cdc003ef0 0x7f9cdc0047c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800c2c0 tx=0x7f9cdc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf2d9d640 1 -- 172.21.15.124:0/2776620134 reap_dead start 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/2776620134 shutdown_connections 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/2776620134 >> 172.21.15.124:0/2776620134 conn(0x7f9cf4072020 msgr2=0x7f9cf4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/2776620134 shutdown_connections 2023-12-13T23:55:12.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.879+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/2776620134 wait complete. 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.880+0000 7f9cf8fc4640 1 Processor -- start 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.880+0000 7f9cf8fc4640 1 -- start start 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cdc003ef0 0x7f9cf41581e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 0x7f9cf41546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40cd720 con 0x7f9cf41527f0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf4154c20 con 0x7f9cf41537a0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf8fc4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf40c83d0 con 0x7f9cdc003ef0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48604/0 (socket says 172.21.15.124:48604) 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 -- 172.21.15.124:0/3016820365 learned_addr learned my addr 172.21.15.124:0/3016820365 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 0x7f9cf41546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf259c640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cdc003ef0 0x7f9cf41581e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 -- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cdc003ef0 msgr2=0x7f9cf41581e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cdc003ef0 0x7f9cf41581e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 -- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 msgr2=0x7f9cf41546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 0x7f9cf41546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 -- 172.21.15.124:0/3016820365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf40c7dc0 con 0x7f9cf41527f0 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf1d9b640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 0x7f9cf41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:12.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9cf2d9d640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9cf408c510 tx=0x7f9cec011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.881+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec011ee0 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.882+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cec004240 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.882+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf4154e50 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.882+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cec00f710 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.882+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf4155130 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.883+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf4069400 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.883+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9cec00f870 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.883+0000 7f9ce2ffd640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 0x7f9cd406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.883+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9cec089a80 con 0x7f9cf41527f0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.884+0000 7f9cf259c640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 0x7f9cd406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:12.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.884+0000 7f9cf259c640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 0x7f9cd406d680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4007440 tx=0x7f9ce400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:12.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:12.888+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9cec0547b0 con 0x7f9cf41527f0 2023-12-13T23:55:13.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.057+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cf40c8510 con 0x7f9cf41527f0 2023-12-13T23:55:13.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.058+0000 7f9ce2ffd640 1 -- 172.21.15.124:0/3016820365 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f9cec006c30 con 0x7f9cf41527f0 2023-12-13T23:55:13.057 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:13.057 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":20,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:12.125273+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 msgr2=0x7f9cd406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 0x7f9cd406d680 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4007440 tx=0x7f9ce400b040 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 msgr2=0x7f9cf4155c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f9cf408c510 tx=0x7f9cec011cf0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 shutdown_connections 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9cd406b1c0 0x7f9cd406d680 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9cf41537a0 0x7f9cf41546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9cf41527f0 0x7f9cf4155c80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 --2- 172.21.15.124:0/3016820365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9cdc003ef0 0x7f9cf41581e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 >> 172.21.15.124:0/3016820365 conn(0x7f9cf4072020 msgr2=0x7f9cf40d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 shutdown_connections 2023-12-13T23:55:13.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.060+0000 7f9cf8fc4640 1 -- 172.21.15.124:0/3016820365 wait complete. 2023-12-13T23:55:13.070 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:13.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.171+0000 7f9fcbd0f640 1 Processor -- start 2023-12-13T23:55:13.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.171+0000 7f9fcbd0f640 1 -- start start 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fcbd0f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9fc40b1400 0x7f9fc415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fcbd0f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fcbd0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fc415e2a0 con 0x7f9fc4143490 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fcbd0f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fc415b520 con 0x7f9fc40b1400 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fcbd0f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc4143870 con 0x7f9fc415e730 2023-12-13T23:55:13.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fca2ac640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc92aa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9fc40b1400 0x7f9fc415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc9aab640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9fc4143490 0x7f9fc40b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:59228/0 (socket says 172.21.15.124:59228) 2023-12-13T23:55:13.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc9aab640 1 -- 172.21.15.124:0/3991656025 learned_addr learned my addr 172.21.15.124:0/3991656025 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 msgr2=0x7f9fc415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc92aa640 1 --2- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc415aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3991656025 >> v1:172.21.15.203:6789/0 conn(0x7f9fc4143490 legacy=0x7f9fc40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.172+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3991656025 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fc415b240 con 0x7f9fc40b1400 2023-12-13T23:55:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.173+0000 7f9fc92aa640 1 --2- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc415a4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40097a0 tx=0x7f9fb402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc824e4f6bc0c428 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.173+0000 7f9fc8aa9640 1 -- 172.21.15.124:0/3991656025 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4031830 con 0x7f9fc40b1400 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.173+0000 7f9fc8aa9640 1 -- 172.21.15.124:0/3991656025 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb4031990 con 0x7f9fc40b1400 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.173+0000 7f9fc8aa9640 1 -- 172.21.15.124:0/3991656025 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb4031d20 con 0x7f9fc40b1400 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 msgr2=0x7f9fc415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc415a4e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9fb40097a0 tx=0x7f9fb402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3991656025 shutdown_connections 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3991656025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc415a4e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3991656025 >> 172.21.15.124:0/3991656025 conn(0x7f9fc406a4d0 msgr2=0x7f9fc406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3991656025 shutdown_connections 2023-12-13T23:55:13.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.174+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3991656025 wait complete. 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.175+0000 7f9fcbd0f640 1 Processor -- start 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.175+0000 7f9fcbd0f640 1 -- start start 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc40a8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc413f190 con 0x7f9fc4143490 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc413e580 con 0x7f9fc40b1400 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fcbd0f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fc413e860 con 0x7f9fc415e730 2023-12-13T23:55:13.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48606/0 (socket says 172.21.15.124:48606) 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3805739292 learned_addr learned my addr 172.21.15.124:0/3805739292 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc9aab640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fca2ac640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc40a8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 msgr2=0x7f9fc40a8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc40a8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 msgr2=0x7f9fc413d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc413d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 -- 172.21.15.124:0/3805739292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4009400 con 0x7f9fc4143490 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fca2ac640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc40a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.176+0000 7f9fc92aa640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9fb402bee0 tx=0x7f9fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.177+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb403a6f0 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.177+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb403a870 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.177+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fc413eb40 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.177+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb403ac20 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.177+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fc413e230 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.178+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fc413f300 con 0x7f9fc4143490 2023-12-13T23:55:13.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.178+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9fb404c3b0 con 0x7f9fc4143490 2023-12-13T23:55:13.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.178+0000 7f9fba7fc640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 0x7f9fa806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.178+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9fb40ab5b0 con 0x7f9fc4143490 2023-12-13T23:55:13.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.179+0000 7f9fc9aab640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 0x7f9fa806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.179+0000 7f9fc9aab640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 0x7f9fa806d680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0005880 tx=0x7f9fc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.183+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9fb40764b0 con 0x7f9fc4143490 2023-12-13T23:55:13.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.349+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fc4069400 con 0x7f9fc4143490 2023-12-13T23:55:13.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.350+0000 7f9fba7fc640 1 -- 172.21.15.124:0/3805739292 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16579 (secure 0 0 0) 0x7f9fb4079ad0 con 0x7f9fc4143490 2023-12-13T23:55:13.348 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:13.349 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":20,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:12.125273+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:13.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 msgr2=0x7f9fa806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 0x7f9fa806d680 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9fc0005880 tx=0x7f9fc0009040 comp rx=0 tx=0).stop 2023-12-13T23:55:13.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 msgr2=0x7f9fc40a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9fb402bee0 tx=0x7f9fb40046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 shutdown_connections 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9fa806b1c0 0x7f9fa806d680 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9fc415e730 0x7f9fc40a8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9fc4143490 0x7f9fc40a84d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 --2- 172.21.15.124:0/3805739292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9fc40b1400 0x7f9fc413d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 >> 172.21.15.124:0/3805739292 conn(0x7f9fc406a4d0 msgr2=0x7f9fc40b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 shutdown_connections 2023-12-13T23:55:13.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.352+0000 7f9fcbd0f640 1 -- 172.21.15.124:0/3805739292 wait complete. 2023-12-13T23:55:13.361 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-13T23:55:13.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.465+0000 7fa90fc75640 1 Processor -- start 2023-12-13T23:55:13.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.465+0000 7fa90fc75640 1 -- start start 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90fc75640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa908160f70 0x7fa9080cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90fc75640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9080ce480 0x7fa9080d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90fc75640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9081614e0 con 0x7fa90815d2d0 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90fc75640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9081617c0 con 0x7fa9080ce480 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90fc75640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa908161aa0 con 0x7fa908160f70 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90e212640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9080ce480 0x7fa9080d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90e212640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9080ce480 0x7fa9080d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48608/0 (socket says 172.21.15.124:48608) 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90da11640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa90815d2d0 0x7fa9080b8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46662/0 (socket says 172.21.15.124:46662) 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.466+0000 7fa90e212640 1 -- 172.21.15.124:0/865723795 learned_addr learned my addr 172.21.15.124:0/865723795 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:13.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90d210640 1 --2- 172.21.15.124:0/865723795 >> v2:172.21.15.203:3300/0 conn(0x7fa908160f70 0x7fa9080cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90e212640 1 -- 172.21.15.124:0/865723795 >> v2:172.21.15.203:3300/0 conn(0x7fa908160f70 msgr2=0x7fa9080cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:13.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90e212640 1 --2- 172.21.15.124:0/865723795 >> v2:172.21.15.203:3300/0 conn(0x7fa908160f70 0x7fa9080cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90ca0f640 1 -- 172.21.15.124:0/865723795 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836087086 0 0) 0x7fa9081614e0 con 0x7fa90815d2d0 2023-12-13T23:55:13.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90e212640 1 -- 172.21.15.124:0/865723795 >> v1:172.21.15.124:6789/0 conn(0x7fa90815d2d0 legacy=0x7fa9080b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90e212640 1 -- 172.21.15.124:0/865723795 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa908161d80 con 0x7fa9080ce480 2023-12-13T23:55:13.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90e212640 1 --2- 172.21.15.124:0/865723795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080d1af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9080cdb80 tx=0x7fa90002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8122ea8a3220023 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90ca0f640 1 -- 172.21.15.124:0/865723795 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900030860 con 0x7fa9080ce480 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90ca0f640 1 -- 172.21.15.124:0/865723795 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9000309c0 con 0x7fa9080ce480 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.467+0000 7fa90ca0f640 1 -- 172.21.15.124:0/865723795 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa900030d50 con 0x7fa9080ce480 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 -- 172.21.15.124:0/865723795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 msgr2=0x7fa9080d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 --2- 172.21.15.124:0/865723795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080d1af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa9080cdb80 tx=0x7fa90002b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 -- 172.21.15.124:0/865723795 shutdown_connections 2023-12-13T23:55:13.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 --2- 172.21.15.124:0/865723795 >> v2:172.21.15.203:3300/0 conn(0x7fa908160f70 0x7fa9080cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 --2- 172.21.15.124:0/865723795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080d1af0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 -- 172.21.15.124:0/865723795 >> 172.21.15.124:0/865723795 conn(0x7fa908072230 msgr2=0x7fa9080cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 -- 172.21.15.124:0/865723795 shutdown_connections 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.468+0000 7fa90fc75640 1 -- 172.21.15.124:0/865723795 wait complete. 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.469+0000 7fa90fc75640 1 Processor -- start 2023-12-13T23:55:13.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.469+0000 7fa90fc75640 1 -- start start 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa90815d2d0 0x7fa90815fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908160f70 0x7fa9081559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080c9be0 con 0x7fa9080ce480 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa908160ba0 con 0x7fa908160f70 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90fc75640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9080b39d0 con 0x7fa90815d2d0 2023-12-13T23:55:13.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48614/0 (socket says 172.21.15.124:48614) 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 -- 172.21.15.124:0/3787398032 learned_addr learned my addr 172.21.15.124:0/3787398032 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 -- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa90815d2d0 msgr2=0x7fa90815fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa90815d2d0 0x7fa90815fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 -- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908160f70 msgr2=0x7fa9081559a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908160f70 0x7fa9081559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 -- 172.21.15.124:0/3787398032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9000094d0 con 0x7fa9080ce480 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.470+0000 7fa90da11640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c8c0 tx=0x7fa90400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.471+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90400a640 con 0x7fa9080ce480 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.471+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90400a7c0 con 0x7fa9080ce480 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.471+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9081605e0 con 0x7fa9080ce480 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.471+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90400ab70 con 0x7fa9080ce480 2023-12-13T23:55:13.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.471+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9080b3660 con 0x7fa9080ce480 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.472+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9080b34c0 con 0x7fa9080ce480 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.472+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa90400acd0 con 0x7fa9080ce480 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.472+0000 7fa8fe7fc640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 0x7fa8ec06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.473+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa904089100 con 0x7fa9080ce480 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.473+0000 7fa90d210640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 0x7fa8ec06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.473+0000 7fa90d210640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 0x7fa8ec06d680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9080ca000 tx=0x7fa8f8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.477+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa9080b34c0 con 0x7fa9080ce480 2023-12-13T23:55:13.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.679+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fa9080b34c0 con 0x7fa9080ce480 2023-12-13T23:55:13.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.763+0000 7fa8fe7fc640 1 -- 172.21.15.124:0/3787398032 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7fa9080b34c0 con 0x7fa9080ce480 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 msgr2=0x7fa8ec06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 0x7fa8ec06d680 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa9080ca000 tx=0x7fa8f8008040 comp rx=0 tx=0).stop 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 msgr2=0x7fa9080c96a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa90400c8c0 tx=0x7fa90400cd80 comp rx=0 tx=0).stop 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 shutdown_connections 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa8ec06b1c0 0x7fa8ec06d680 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa908160f70 0x7fa9081559a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa90815d2d0 0x7fa90815fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 --2- 172.21.15.124:0/3787398032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9080ce480 0x7fa9080c96a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 >> 172.21.15.124:0/3787398032 conn(0x7fa908072230 msgr2=0x7fa9080b8660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 shutdown_connections 2023-12-13T23:55:13.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.765+0000 7fa90fc75640 1 -- 172.21.15.124:0/3787398032 wait complete. 2023-12-13T23:55:13.774 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-13T23:55:13.775 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-13T23:55:13.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 Processor -- start 2023-12-13T23:55:13.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 -- start start 2023-12-13T23:55:13.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f86a40ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b49f0 con 0x7f86a4150560 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b3560 con 0x7f86a4150940 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.879+0000 7f86aa49b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40b3de0 con 0x7f86a415d910 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f86a4150560 0x7f86a40b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46678/0 (socket says 172.21.15.124:46678) 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a3fff640 1 -- 172.21.15.124:0/2319558817 learned_addr learned my addr 172.21.15.124:0/2319558817 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a8a38640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f86a40ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385874112 0 0) 0x7f86a40b49f0 con 0x7f86a4150560 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f868c003520 con 0x7f86a4150560 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835369486 0 0) 0x7f86a40b3560 con 0x7f86a4150940 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a40b49f0 con 0x7f86a4150940 2023-12-13T23:55:13.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991328761 0 0) 0x7f868c003520 con 0x7f86a4150560 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 msgr2=0x7f86a40ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f86a40ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 >> v1:172.21.15.203:6789/0 conn(0x7f86a4150940 legacy=0x7f86a415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40b40c0 con 0x7f86a4150560 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8694001670 con 0x7f86a4150560 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 >> v1:172.21.15.124:6789/0 conn(0x7f86a4150560 legacy=0x7f86a40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f868c003b30 0x7f868c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f868c0047c0 0x7f868c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f868c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40b3de0 con 0x7f868c0047c0 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c00ba30 con 0x7f868c003b30 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f868c00bbf0 con 0x7f86a415d910 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a37fe640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f868c0047c0 0x7f868c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a3fff640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f868c003b30 0x7f868c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.880+0000 7f86a8a38640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f868c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 -- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 msgr2=0x7f868c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f86a415d910 0x7f868c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 -- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f868c003b30 msgr2=0x7f868c003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f868c003b30 0x7f868c003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 -- 172.21.15.124:0/2319558817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868c003520 con 0x7f868c0047c0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a37fe640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f868c0047c0 0x7f868c004c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86980018a0 tx=0x7f868c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a49609985bff3d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86980052d0 con 0x7f868c0047c0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8698005430 con 0x7f868c0047c0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.881+0000 7f86a2ffd640 1 -- 172.21.15.124:0/2319558817 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8698003810 con 0x7f868c0047c0 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 -- 172.21.15.124:0/2319558817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f868c0047c0 msgr2=0x7f868c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 --2- 172.21.15.124:0/2319558817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f868c0047c0 0x7f868c004c40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f86980018a0 tx=0x7f868c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86a8a38640 1 -- 172.21.15.124:0/2319558817 reap_dead start 2023-12-13T23:55:13.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 -- 172.21.15.124:0/2319558817 shutdown_connections 2023-12-13T23:55:13.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 -- 172.21.15.124:0/2319558817 >> 172.21.15.124:0/2319558817 conn(0x7f86a406a4d0 msgr2=0x7f86a406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:13.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 -- 172.21.15.124:0/2319558817 shutdown_connections 2023-12-13T23:55:13.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.882+0000 7f86aa49b640 1 -- 172.21.15.124:0/2319558817 wait complete. 2023-12-13T23:55:13.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.883+0000 7f86aa49b640 1 Processor -- start 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.883+0000 7f86aa49b640 1 -- start start 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 0x7f86a41ee600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 0x7f86a40aacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a40a8a30 con 0x7f86a41e3e80 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41eeb40 con 0x7f86a41e3a00 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86aa49b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86a41e4450 con 0x7f868c0047c0 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48636/0 (socket says 172.21.15.124:48636) 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 -- 172.21.15.124:0/4252829073 learned_addr learned my addr 172.21.15.124:0/4252829073 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a37fe640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 0x7f86a40aacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a3fff640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 0x7f86a41ee600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 -- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 msgr2=0x7f86a41ee600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 0x7f86a41ee600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 -- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 msgr2=0x7f86a40aacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 0x7f86a40aacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a8a38640 1 -- 172.21.15.124:0/4252829073 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a41e4730 con 0x7f86a41e3e80 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a3fff640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 0x7f86a41ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.884+0000 7f86a37fe640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 0x7f86a40aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86a8a38640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86a41577e0 tx=0x7f869c011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c00e040 con 0x7f86a41e3e80 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f869c00f040 con 0x7f86a41e3e80 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a40a8650 con 0x7f86a41e3e80 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f869c007780 con 0x7f86a41e3e80 2023-12-13T23:55:13.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.885+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a414ffe0 con 0x7f86a41e3e80 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.886+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a40a8b70 con 0x7f86a41e3e80 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.886+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f869c0078e0 con 0x7f86a41e3e80 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.886+0000 7f86a0ff9640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 0x7f869006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.886+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f869c08bb20 con 0x7f86a41e3e80 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.887+0000 7f86a3fff640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 0x7f869006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.887+0000 7f86a3fff640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 0x7f869006d6b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8694001050 tx=0x7f8694009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:13.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:13.891+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f86a40a8b70 con 0x7f86a41e3e80 2023-12-13T23:55:14.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.091+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f86a40a8b70 con 0x7f86a41e3e80 2023-12-13T23:55:14.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.768+0000 7f86a0ff9640 1 -- 172.21.15.124:0/4252829073 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7f86a40a8b70 con 0x7f86a41e3e80 2023-12-13T23:55:14.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 msgr2=0x7f869006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 0x7f869006d6b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8694001050 tx=0x7f8694009040 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 msgr2=0x7f86a40ab200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f86a41577e0 tx=0x7f869c011790 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 shutdown_connections 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f869006b1f0 0x7f869006d6b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f86a41e3a00 0x7f86a40aacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f86a41e3e80 0x7f86a40ab200 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 --2- 172.21.15.124:0/4252829073 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f868c0047c0 0x7f86a41ee600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 >> 172.21.15.124:0/4252829073 conn(0x7f86a406a4d0 msgr2=0x7f86a40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 shutdown_connections 2023-12-13T23:55:14.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.770+0000 7f86aa49b640 1 -- 172.21.15.124:0/4252829073 wait complete. 2023-12-13T23:55:14.780 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-13T23:55:14.780 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.884+0000 7f473a0b4640 1 Processor -- start 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.884+0000 7f473a0b4640 1 -- start start 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f473a0b4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4734150940 0x7f473415d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f473a0b4640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340b49f0 con 0x7f4734150560 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f473a0b4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47340b3560 con 0x7f473415dec0 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f473a0b4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47340b3de0 con 0x7f4734150940 2023-12-13T23:55:14.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47337fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f4734150560 0x7f47340b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46688/0 (socket says 172.21.15.124:46688) 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47337fe640 1 -- 172.21.15.124:0/2230594685 learned_addr learned my addr 172.21.15.124:0/2230594685 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f4732ffd640 1 --2- 172.21.15.124:0/2230594685 >> v2:172.21.15.203:3300/0 conn(0x7f4734150940 0x7f473415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831019841 0 0) 0x7f47340b49f0 con 0x7f4734150560 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f471c004630 con 0x7f4734150560 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1687611221 0 0) 0x7f471c004630 con 0x7f4734150560 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 >> v2:172.21.15.203:3300/0 conn(0x7f4734150940 msgr2=0x7f473415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 --2- 172.21.15.124:0/2230594685 >> v2:172.21.15.203:3300/0 conn(0x7f4734150940 0x7f473415d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 >> v1:172.21.15.203:6789/0 conn(0x7f473415dec0 legacy=0x7f47340ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47340b40c0 con 0x7f4734150560 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f4724001670 con 0x7f4734150560 2023-12-13T23:55:14.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 >> v1:172.21.15.124:6789/0 conn(0x7f4734150560 legacy=0x7f47340b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c004a80 0x7f471c004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471c0058d0 0x7f471c005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 0x7f471c00ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47340b3de0 con 0x7f471c006290 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c00d130 con 0x7f471c0058d0 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c00d2f0 con 0x7f471c004a80 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f4733fff640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 0x7f471c00ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f4732ffd640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471c0058d0 0x7f471c005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.885+0000 7f47337fe640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c004a80 0x7f471c004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 -- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c004a80 msgr2=0x7f471c004ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c004a80 0x7f471c004ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 -- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471c0058d0 msgr2=0x7f471c005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f471c0058d0 0x7f471c005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 -- 172.21.15.124:0/2230594685 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c004630 con 0x7f471c006290 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 -- 172.21.15.124:0/2230594685 reap_dead start 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f4733fff640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 0x7f471c00ca90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f472c001e50 tx=0x7f471c006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf10d89639f19c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:14.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c003b70 con 0x7f471c006290 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472c003cf0 con 0x7f471c006290 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.886+0000 7f47327fc640 1 -- 172.21.15.124:0/2230594685 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472c005310 con 0x7f471c006290 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 -- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 msgr2=0x7f471c00ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 0x7f471c00ca90 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f472c001e50 tx=0x7f471c006900 comp rx=0 tx=0).stop 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 -- 172.21.15.124:0/2230594685 shutdown_connections 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2230594685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f471c006290 0x7f471c00ca90 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 -- 172.21.15.124:0/2230594685 >> 172.21.15.124:0/2230594685 conn(0x7f473406a4d0 msgr2=0x7f473406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 -- 172.21.15.124:0/2230594685 shutdown_connections 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.887+0000 7f473a0b4640 1 -- 172.21.15.124:0/2230594685 wait complete. 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.888+0000 7f473a0b4640 1 Processor -- start 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.888+0000 7f473a0b4640 1 -- start start 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c006290 0x7f473414cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4734158340 0x7f47341587b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473415b780 con 0x7f473415a520 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4734158cf0 con 0x7f4734158340 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f473a0b4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f473415c9e0 con 0x7f471c006290 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48654/0 (socket says 172.21.15.124:48654) 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 -- 172.21.15.124:0/2292393997 learned_addr learned my addr 172.21.15.124:0/2292393997 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4733fff640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4734158340 0x7f47341587b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f47337fe640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c006290 0x7f473414cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 -- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c006290 msgr2=0x7f473414cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c006290 0x7f473414cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 -- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4734158340 msgr2=0x7f47341587b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4734158340 0x7f47341587b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:14.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.889+0000 7f4732ffd640 1 -- 172.21.15.124:0/2292393997 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473413bed0 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f4732ffd640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4728011aa0 tx=0x7f4728011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4728016070 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472800eb50 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f472800ecc0 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473415a9a0 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.890+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473415ba10 con 0x7f473415a520 2023-12-13T23:55:14.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.891+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4734071ff0 con 0x7f473415a520 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.891+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f472801a410 con 0x7f473415a520 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.892+0000 7f471bfff640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 0x7f472006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.892+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f472808b460 con 0x7f473415a520 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.892+0000 7f47337fe640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 0x7f472006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.892+0000 7f47337fe640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 0x7f472006d680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f47340b0b90 tx=0x7f4724009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:14.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:14.896+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f47280562d0 con 0x7f473415a520 2023-12-13T23:55:15.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.098+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4734069400 con 0x7f473415a520 2023-12-13T23:55:15.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.098+0000 7f471bfff640 1 -- 172.21.15.124:0/2292393997 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f47280598f0 con 0x7f473415a520 2023-12-13T23:55:15.097 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:15.097 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:15.097 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:15.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 msgr2=0x7f472006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 0x7f472006d680 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f47340b0b90 tx=0x7f4724009040 comp rx=0 tx=0).stop 2023-12-13T23:55:15.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 msgr2=0x7f473413b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4728011aa0 tx=0x7f4728011f60 comp rx=0 tx=0).stop 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 shutdown_connections 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f472006b1c0 0x7f472006d680 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4734158340 0x7f47341587b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f473415a520 0x7f473413b990 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 --2- 172.21.15.124:0/2292393997 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f471c006290 0x7f473414cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 >> 172.21.15.124:0/2292393997 conn(0x7f473406a4d0 msgr2=0x7f473415d160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 shutdown_connections 2023-12-13T23:55:15.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.100+0000 7f473a0b4640 1 -- 172.21.15.124:0/2292393997 wait complete. 2023-12-13T23:55:15.110 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:15.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.212+0000 7f58b0801640 1 Processor -- start 2023-12-13T23:55:15.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.212+0000 7f58b0801640 1 -- start start 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58b0801640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f58ac160c30 0x7f58ac15e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58b0801640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac06f450 con 0x7f58ac0b2580 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58b0801640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac1612f0 con 0x7f58ac06f070 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58b0801640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac161b90 con 0x7f58ac160c30 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58aa59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f58ac160c30 0x7f58ac15e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58ab59e640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f58ac0b2580 0x7f58ac06b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46696/0 (socket says 172.21.15.124:46696) 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58ab59e640 1 -- 172.21.15.124:0/1817698996 learned_addr learned my addr 172.21.15.124:0/1817698996 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:15.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58aa59c640 1 -- 172.21.15.124:0/1817698996 >> v1:172.21.15.203:6789/0 conn(0x7f58ac06f070 legacy=0x7f58ac0ae1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58aa59c640 1 -- 172.21.15.124:0/1817698996 >> v1:172.21.15.124:6789/0 conn(0x7f58ac0b2580 legacy=0x7f58ac06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.213+0000 7f58aa59c640 1 -- 172.21.15.124:0/1817698996 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac1615d0 con 0x7f58ac160c30 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58aa59c640 1 --2- 172.21.15.124:0/1817698996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac160c30 0x7f58ac15e0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e88c1d54ef04ef server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58a9d9b640 1 -- 172.21.15.124:0/1817698996 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0030710 con 0x7f58ac160c30 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58a9d9b640 1 -- 172.21.15.124:0/1817698996 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a002e050 con 0x7f58ac160c30 2023-12-13T23:55:15.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58a9d9b640 1 -- 172.21.15.124:0/1817698996 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0030a90 con 0x7f58ac160c30 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58b0801640 1 -- 172.21.15.124:0/1817698996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac160c30 msgr2=0x7f58ac15e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.214+0000 7f58b0801640 1 --2- 172.21.15.124:0/1817698996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac160c30 0x7f58ac15e0c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f58a00097a0 tx=0x7f58a002b790 comp rx=0 tx=0).stop 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.215+0000 7f58b0801640 1 -- 172.21.15.124:0/1817698996 shutdown_connections 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.215+0000 7f58b0801640 1 --2- 172.21.15.124:0/1817698996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac160c30 0x7f58ac15e0c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.215+0000 7f58b0801640 1 -- 172.21.15.124:0/1817698996 >> 172.21.15.124:0/1817698996 conn(0x7f58ac070a90 msgr2=0x7f58ac070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.215+0000 7f58b0801640 1 -- 172.21.15.124:0/1817698996 shutdown_connections 2023-12-13T23:55:15.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.215+0000 7f58b0801640 1 -- 172.21.15.124:0/1817698996 wait complete. 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.216+0000 7f58b0801640 1 Processor -- start 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.216+0000 7f58b0801640 1 -- start start 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 0x7f58ac14c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 0x7f58ac15ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1f34b0 con 0x7f58ac06f070 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac14c550 con 0x7f58ac160c30 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac14fa50 con 0x7f58ac0b2580 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48676/0 (socket says 172.21.15.124:48676) 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 -- 172.21.15.124:0/556955151 learned_addr learned my addr 172.21.15.124:0/556955151 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aa59c640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 0x7f58ac14c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58ab59e640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 0x7f58ac15ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 -- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 msgr2=0x7f58ac14c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 0x7f58ac14c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 -- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 msgr2=0x7f58ac15ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 0x7f58ac15ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 -- 172.21.15.124:0/556955151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a0009400 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aa59c640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 0x7f58ac14c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58ab59e640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 0x7f58ac15ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58aad9d640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f589c00c730 tx=0x7f589c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c003b70 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c003cf0 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00aa90 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.217+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac1f2b10 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.218+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac1f30d0 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.218+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac069320 con 0x7f58ac06f070 2023-12-13T23:55:15.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.219+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f589c00abf0 con 0x7f58ac06f070 2023-12-13T23:55:15.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.219+0000 7f58937fe640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 0x7f589806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.219+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f589c0897c0 con 0x7f58ac06f070 2023-12-13T23:55:15.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.220+0000 7f58aa59c640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 0x7f589806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.220+0000 7f58aa59c640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 0x7f589806d680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034000 tx=0x7f58a002f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.223+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f589c054580 con 0x7f58ac06f070 2023-12-13T23:55:15.390 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.392+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58ac153600 con 0x7f58ac06f070 2023-12-13T23:55:15.391 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.393+0000 7f58937fe640 1 -- 172.21.15.124:0/556955151 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f589c003e60 con 0x7f58ac06f070 2023-12-13T23:55:15.391 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:15.391 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.394+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 msgr2=0x7f589806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.394+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 0x7f589806d680 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58a0034000 tx=0x7f58a002f000 comp rx=0 tx=0).stop 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.394+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 msgr2=0x7f58ac14d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.394+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f589c00c730 tx=0x7f589c00cbf0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 shutdown_connections 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f589806b1c0 0x7f589806d680 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58ac160c30 0x7f58ac15ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58ac0b2580 0x7f58ac14c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 --2- 172.21.15.124:0/556955151 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58ac06f070 0x7f58ac14d0d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 >> 172.21.15.124:0/556955151 conn(0x7f58ac070a90 msgr2=0x7f58ac0b0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 shutdown_connections 2023-12-13T23:55:15.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.395+0000 7f58b0801640 1 -- 172.21.15.124:0/556955151 wait complete. 2023-12-13T23:55:15.404 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-13T23:55:15.404 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:15.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.505+0000 7f6041f4c640 1 Processor -- start 2023-12-13T23:55:15.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.505+0000 7f6041f4c640 1 -- start start 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f6041f4c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f603c06ea50 0x7f603c06ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f6041f4c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f603c0d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f6041f4c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f603c161d20 con 0x7f603c160fb0 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f6041f4c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f603c06f370 con 0x7f603c06ea50 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f6041f4c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15e760 con 0x7f603c0d0ed0 2023-12-13T23:55:15.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603affd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f603c160fb0 0x7f603c161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46698/0 (socket says 172.21.15.124:46698) 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603affd640 1 -- 172.21.15.124:0/3103429474 learned_addr learned my addr 172.21.15.124:0/3103429474 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603bfff640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f603c0d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603b7fe640 1 --2- 172.21.15.124:0/3103429474 >> v2:172.21.15.203:3300/0 conn(0x7f603c06ea50 0x7f603c06ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 831175412 0 0) 0x7f603c161d20 con 0x7f603c160fb0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6024003520 con 0x7f603c160fb0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.506+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 496846609 0 0) 0x7f6024003520 con 0x7f603c160fb0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 msgr2=0x7f603c0d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f603c0d1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 >> v2:172.21.15.203:3300/0 conn(0x7f603c06ea50 msgr2=0x7f603c06ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 --2- 172.21.15.124:0/3103429474 >> v2:172.21.15.203:3300/0 conn(0x7f603c06ea50 0x7f603c06ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c15d9a0 con 0x7f603c160fb0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f6030001670 con 0x7f603c160fb0 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 >> v1:172.21.15.124:6789/0 conn(0x7f603c160fb0 legacy=0x7f603c161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6024003970 0x7f6024003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60240047c0 0x7f6024004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f6024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c06f370 con 0x7f60240047c0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c15e760 con 0x7f6024003970 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602400b980 con 0x7f603c0d0ed0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60240047c0 0x7f6024004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603b7fe640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6024003970 0x7f6024003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603bfff640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f6024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 -- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 msgr2=0x7f6024005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f603c0d0ed0 0x7f6024005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 -- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6024003970 msgr2=0x7f6024003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6024003970 0x7f6024003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.507+0000 7f603affd640 1 -- 172.21.15.124:0/3103429474 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6024003520 con 0x7f60240047c0 2023-12-13T23:55:15.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f603affd640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60240047c0 0x7f6024004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f603000c290 tx=0x7f60240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e22eb9794b2ccb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6030005ad0 con 0x7f60240047c0 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6030005c30 con 0x7f60240047c0 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f603a7fc640 1 -- 172.21.15.124:0/3103429474 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603002c8d0 con 0x7f60240047c0 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 -- 172.21.15.124:0/3103429474 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60240047c0 msgr2=0x7f6024004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 --2- 172.21.15.124:0/3103429474 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60240047c0 0x7f6024004c40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f603000c290 tx=0x7f60240057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f603bfff640 1 -- 172.21.15.124:0/3103429474 reap_dead start 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 -- 172.21.15.124:0/3103429474 shutdown_connections 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 -- 172.21.15.124:0/3103429474 >> 172.21.15.124:0/3103429474 conn(0x7f603c06a6b0 msgr2=0x7f603c06aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 -- 172.21.15.124:0/3103429474 shutdown_connections 2023-12-13T23:55:15.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.508+0000 7f6041f4c640 1 -- 172.21.15.124:0/3103429474 wait complete. 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.509+0000 7f6041f4c640 1 Processor -- start 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 -- start start 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 0x7f603c13ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 0x7f603c1f3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c1f3f90 con 0x7f603c13f510 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603c13fdb0 con 0x7f603c1f35d0 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f6041f4c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603c06c700 con 0x7f60240047c0 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f603affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f603affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48700/0 (socket says 172.21.15.124:48700) 2023-12-13T23:55:15.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.510+0000 7f603affd640 1 -- 172.21.15.124:0/1120229144 learned_addr learned my addr 172.21.15.124:0/1120229144 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603bfff640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 0x7f603c1f3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 -- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 msgr2=0x7f603c13ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603b7fe640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 0x7f603c13ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 0x7f603c13ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 -- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 msgr2=0x7f603c1f3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 0x7f603c1f3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 -- 172.21.15.124:0/1120229144 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f603c06c9e0 con 0x7f603c13f510 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603bfff640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 0x7f603c1f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603b7fe640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 0x7f603c13ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f603affd640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f603000c3b0 tx=0x7f6030000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603002cce0 con 0x7f603c13f510 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603002ce60 con 0x7f603c13f510 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603c06ccc0 con 0x7f603c13f510 2023-12-13T23:55:15.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60300072b0 con 0x7f603c13f510 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.511+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603c06d280 con 0x7f603c13f510 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.512+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f603c069400 con 0x7f603c13f510 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.513+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6030032070 con 0x7f603c13f510 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.513+0000 7f6023fff640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 0x7f602806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.513+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f603003c040 con 0x7f603c13f510 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.513+0000 7f603b7fe640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 0x7f602806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.514+0000 7f603b7fe640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 0x7f602806d680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f602c009a40 tx=0x7f602c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.517+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6030075c20 con 0x7f603c13f510 2023-12-13T23:55:15.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.681+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f603c15e240 con 0x7f602806b1c0 2023-12-13T23:55:15.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.688+0000 7f6023fff640 1 -- 172.21.15.124:0/1120229144 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249762 (secure 0 0 0) 0x7f602c013a20 con 0x7f602806b1c0 2023-12-13T23:55:15.689 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:15.690 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 msgr2=0x7f602806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 0x7f602806d680 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f602c009a40 tx=0x7f602c008040 comp rx=0 tx=0).stop 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 msgr2=0x7f603c1f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f603000c3b0 tx=0x7f6030000fe0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 shutdown_connections 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f602806b1c0 0x7f602806d680 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f603c1f35d0 0x7f603c1f3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f603c13f510 0x7f603c1f3090 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 --2- 172.21.15.124:0/1120229144 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60240047c0 0x7f603c13ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 >> 172.21.15.124:0/1120229144 conn(0x7f603c06a6b0 msgr2=0x7f603c0b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 shutdown_connections 2023-12-13T23:55:15.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.691+0000 7f6041f4c640 1 -- 172.21.15.124:0/1120229144 wait complete. 2023-12-13T23:55:15.700 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-12-13T23:55:14.109550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219388,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":2531328,"data_stored":1584048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000792"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-13T23:55:03.774133+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:03.774133+0000","last_peered":"2023-12-13T23:55:03.774133+0000","last_clean":"2023-12-13T23:55:03.774133+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:03.774133+0000","last_undegraded":"2023-12-13T23:55:03.774133+0000","last_fullsized":"2023-12-13T23:55:03.774133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:15.705 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:15.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.842+0000 7f1c132bb640 1 Processor -- start 2023-12-13T23:55:15.841 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 -- start start 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 0x7f1c0c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c0c15e2a0 con 0x7f1c0c143490 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0c15b520 con 0x7f1c0c15e730 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.843+0000 7f1c132bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c143870 con 0x7f1c0c0b1400 2023-12-13T23:55:15.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c11858640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 0x7f1c0c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:51904/0 (socket says 172.21.15.124:51904) 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 -- 172.21.15.124:0/2076066949 learned_addr learned my addr 172.21.15.124:0/2076066949 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 -- 172.21.15.124:0/2076066949 >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 msgr2=0x7f1c0c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 --2- 172.21.15.124:0/2076066949 >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 0x7f1c0c15aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 -- 172.21.15.124:0/2076066949 >> v1:172.21.15.203:6789/0 conn(0x7f1c0c143490 legacy=0x7f1c0c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c10856640 1 -- 172.21.15.124:0/2076066949 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c15b240 con 0x7f1c0c0b1400 2023-12-13T23:55:15.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.844+0000 7f1c11858640 1 --2- 172.21.15.124:0/2076066949 >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 0x7f1c0c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:15.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c10856640 1 --2- 172.21.15.124:0/2076066949 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009870 tx=0x7f1bfc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd13a555b54ca297 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c03fff640 1 -- 172.21.15.124:0/2076066949 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc030840 con 0x7f1c0c0b1400 2023-12-13T23:55:15.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c03fff640 1 -- 172.21.15.124:0/2076066949 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc0309a0 con 0x7f1c0c0b1400 2023-12-13T23:55:15.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c03fff640 1 -- 172.21.15.124:0/2076066949 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc030d30 con 0x7f1c0c0b1400 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 -- 172.21.15.124:0/2076066949 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 msgr2=0x7f1c0c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 --2- 172.21.15.124:0/2076066949 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc009870 tx=0x7f1bfc02b980 comp rx=0 tx=0).stop 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 -- 172.21.15.124:0/2076066949 shutdown_connections 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 --2- 172.21.15.124:0/2076066949 >> v2:172.21.15.203:3300/0 conn(0x7f1c0c15e730 0x7f1c0c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 --2- 172.21.15.124:0/2076066949 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c0b1400 0x7f1c0c15a4e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.845+0000 7f1c132bb640 1 -- 172.21.15.124:0/2076066949 >> 172.21.15.124:0/2076066949 conn(0x7f1c0c06a4d0 msgr2=0x7f1c0c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.846+0000 7f1c132bb640 1 -- 172.21.15.124:0/2076066949 shutdown_connections 2023-12-13T23:55:15.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.846+0000 7f1c132bb640 1 -- 172.21.15.124:0/2076066949 wait complete. 2023-12-13T23:55:15.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 Processor -- start 2023-12-13T23:55:15.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 -- start start 2023-12-13T23:55:15.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0c0b1400 0x7f1c0c13eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c143490 0x7f1c0c159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c13f3f0 con 0x7f1c0c15e730 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c138a70 con 0x7f1c0c0b1400 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.847+0000 7f1c132bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c1395f0 con 0x7f1c0c143490 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48706/0 (socket says 172.21.15.124:48706) 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 -- 172.21.15.124:0/3502091962 learned_addr learned my addr 172.21.15.124:0/3502091962 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c10856640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c143490 0x7f1c0c159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11057640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0c0b1400 0x7f1c0c13eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 -- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c143490 msgr2=0x7f1c0c159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c143490 0x7f1c0c159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 -- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0c0b1400 msgr2=0x7f1c0c13eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0c0b1400 0x7f1c0c13eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:15.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 -- 172.21.15.124:0/3502091962 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c08009400 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c11858640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e880 tx=0x7f1c0800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c08009660 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08004020 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bfc0094d0 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c080043b0 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.848+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c139a80 con 0x7f1c0c15e730 2023-12-13T23:55:15.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.849+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c139760 con 0x7f1c0c15e730 2023-12-13T23:55:15.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.850+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1c0800c940 con 0x7f1c0c15e730 2023-12-13T23:55:15.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.850+0000 7f1c01ffb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 0x7f1bf006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:15.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.850+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c08089900 con 0x7f1c0c15e730 2023-12-13T23:55:15.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.850+0000 7f1c11057640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 0x7f1bf006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:15.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.851+0000 7f1c11057640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 0x7f1bf006d680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c137690 tx=0x7f1c04009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:15.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:15.854+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1c08054660 con 0x7f1c0c15e730 2023-12-13T23:55:16.016 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.017+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c0c15af60 con 0x7f1bf006b1c0 2023-12-13T23:55:16.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.023+0000 7f1c01ffb640 1 -- 172.21.15.124:0/3502091962 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+249762 (secure 0 0 0) 0x7f1c0400fdd0 con 0x7f1bf006b1c0 2023-12-13T23:55:16.024 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:16.025 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 msgr2=0x7f1bf006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 0x7f1bf006d680 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c137690 tx=0x7f1c04009040 comp rx=0 tx=0).stop 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 msgr2=0x7f1c0c159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1c0800e880 tx=0x7f1c0800ed40 comp rx=0 tx=0).stop 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 shutdown_connections 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1bf006b1c0 0x7f1bf006d680 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c0c15e730 0x7f1c0c159a70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1c0c143490 0x7f1c0c159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 --2- 172.21.15.124:0/3502091962 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1c0c0b1400 0x7f1c0c13eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 >> 172.21.15.124:0/3502091962 conn(0x7f1c0c06a4d0 msgr2=0x7f1c0c0af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 shutdown_connections 2023-12-13T23:55:16.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.026+0000 7f1c132bb640 1 -- 172.21.15.124:0/3502091962 wait complete. 2023-12-13T23:55:16.035 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-12-13T23:55:14.109550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":219388,"kb_used_data":2472,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755332,"statfs":{"total":773094113280,"available":772869459968,"internally_reserved":0,"allocated":2531328,"data_stored":1584048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000792"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":161,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-12-13T23:55:03.774133+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:03.774133+0000","last_peered":"2023-12-13T23:55:03.774133+0000","last_clean":"2023-12-13T23:55:03.774133+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:03.774133+0000","last_undegraded":"2023-12-13T23:55:03.774133+0000","last_fullsized":"2023-12-13T23:55:03.774133+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:11.740391+0000","last_change":"2023-12-13T23:55:11.740391+0000","last_active":"2023-12-13T23:55:11.740391+0000","last_peered":"2023-12-13T23:55:11.740391+0000","last_clean":"2023-12-13T23:55:11.740391+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:11.740391+0000","last_undegraded":"2023-12-13T23:55:11.740391+0000","last_fullsized":"2023-12-13T23:55:11.740391+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:55:10.735939+0000","last_change":"2023-12-13T23:55:10.735939+0000","last_active":"2023-12-13T23:55:10.735939+0000","last_peered":"2023-12-13T23:55:10.735939+0000","last_clean":"2023-12-13T23:55:10.735939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:55:10.735939+0000","last_undegraded":"2023-12-13T23:55:10.735939+0000","last_fullsized":"2023-12-13T23:55:10.735939+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27192,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.499}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.436}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27188,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.629}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27184,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344656,"statfs":{"total":96636764160,"available":96608927744,"internally_reserved":0,"allocated":167936,"data_stored":50414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28384,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343456,"statfs":{"total":96636764160,"available":96607698944,"internally_reserved":0,"allocated":761856,"data_stored":640782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:16.039 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:16.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.176+0000 7fdf44cd3640 1 Processor -- start 2023-12-13T23:55:16.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.177+0000 7fdf44cd3640 1 -- start start 2023-12-13T23:55:16.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.177+0000 7fdf44cd3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 0x7fdf400cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.177+0000 7fdf44cd3640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 0x7fdf400c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.177+0000 7fdf44cd3640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf400cf0f0 con 0x7fdf400c63c0 2023-12-13T23:55:16.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf44cd3640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf400c7ee0 con 0x7fdf4015bfc0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf44cd3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400cf330 con 0x7fdf4015bbe0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3ed9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdf400c63c0 0x7fdf4015f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46712/0 (socket says 172.21.15.124:46712) 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3ed9d640 1 -- 172.21.15.124:0/1171042876 learned_addr learned my addr 172.21.15.124:0/1171042876 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3dd9b640 1 --2- 172.21.15.124:0/1171042876 >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 0x7fdf400c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/1171042876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 0x7fdf400cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3d59a640 1 -- 172.21.15.124:0/1171042876 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265151042 0 0) 0x7fdf30001110 con 0x7fdf400c63c0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3d59a640 1 -- 172.21.15.124:0/1171042876 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf28003520 con 0x7fdf400c63c0 2023-12-13T23:55:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3e59c640 1 -- 172.21.15.124:0/1171042876 >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 msgr2=0x7fdf400c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/1171042876 >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 0x7fdf400c5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3e59c640 1 -- 172.21.15.124:0/1171042876 >> v1:172.21.15.124:6789/0 conn(0x7fdf400c63c0 legacy=0x7fdf4015f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3e59c640 1 -- 172.21.15.124:0/1171042876 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf400c7920 con 0x7fdf4015bbe0 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.178+0000 7fdf3dd9b640 1 --2- 172.21.15.124:0/1171042876 >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 0x7fdf400c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.179+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/1171042876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 0x7fdf400cebb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf380097a0 tx=0x7fdf3802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b94a5d941d64d5ae server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.179+0000 7fdf3d59a640 1 -- 172.21.15.124:0/1171042876 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3802e070 con 0x7fdf4015bbe0 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.179+0000 7fdf3d59a640 1 -- 172.21.15.124:0/1171042876 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf380315a0 con 0x7fdf4015bbe0 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.179+0000 7fdf3d59a640 1 -- 172.21.15.124:0/1171042876 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf38031930 con 0x7fdf4015bbe0 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 -- 172.21.15.124:0/1171042876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 msgr2=0x7fdf400cebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/1171042876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 0x7fdf400cebb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdf380097a0 tx=0x7fdf3802b840 comp rx=0 tx=0).stop 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 -- 172.21.15.124:0/1171042876 shutdown_connections 2023-12-13T23:55:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/1171042876 >> v2:172.21.15.203:3300/0 conn(0x7fdf4015bfc0 0x7fdf400c5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/1171042876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf4015bbe0 0x7fdf400cebb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 -- 172.21.15.124:0/1171042876 >> 172.21.15.124:0/1171042876 conn(0x7fdf400b2c10 msgr2=0x7fdf400b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 -- 172.21.15.124:0/1171042876 shutdown_connections 2023-12-13T23:55:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.180+0000 7fdf44cd3640 1 -- 172.21.15.124:0/1171042876 wait complete. 2023-12-13T23:55:16.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.181+0000 7fdf44cd3640 1 Processor -- start 2023-12-13T23:55:16.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.181+0000 7fdf44cd3640 1 -- start start 2023-12-13T23:55:16.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf400c63c0 0x7fdf40154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 0x7fdf400ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400c8a50 con 0x7fdf4015bfc0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf40154550 con 0x7fdf4015bbe0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf400cab80 con 0x7fdf400c63c0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48714/0 (socket says 172.21.15.124:48714) 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 -- 172.21.15.124:0/98148246 learned_addr learned my addr 172.21.15.124:0/98148246 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf400c63c0 0x7fdf40154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3dd9b640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 0x7fdf400ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 -- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf400c63c0 msgr2=0x7fdf40154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf400c63c0 0x7fdf40154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 -- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 msgr2=0x7fdf400ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 0x7fdf400ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 -- 172.21.15.124:0/98148246 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf34009a70 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3dd9b640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 0x7fdf400ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf3ed9d640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c700 tx=0x7fdf3000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30005ab0 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf30005c10 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf30001d50 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.182+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf38009400 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.183+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf400c8390 con 0x7fdf4015bfc0 2023-12-13T23:55:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.183+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf400c8b90 con 0x7fdf4015bfc0 2023-12-13T23:55:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.184+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdf30007070 con 0x7fdf4015bfc0 2023-12-13T23:55:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.184+0000 7fdf2effd640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 0x7fdf2006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.184+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf30088dd0 con 0x7fdf4015bfc0 2023-12-13T23:55:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.185+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 0x7fdf2006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.185+0000 7fdf3e59c640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 0x7fdf2006d680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf38036000 tx=0x7fdf3803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.188+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdf400c8b90 con 0x7fdf4015bfc0 2023-12-13T23:55:16.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.355+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdf4015ed60 con 0x7fdf2006b1c0 2023-12-13T23:55:16.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.362+0000 7fdf2effd640 1 -- 172.21.15.124:0/98148246 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264701 (secure 0 0 0) 0x7fdf380319d0 con 0x7fdf2006b1c0 2023-12-13T23:55:16.363 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:16.364 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:16.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.364+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 msgr2=0x7fdf2006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.364+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 0x7fdf2006d680 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdf38036000 tx=0x7fdf3803d040 comp rx=0 tx=0).stop 2023-12-13T23:55:16.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 msgr2=0x7fdf400c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdf3000c700 tx=0x7fdf3000cbc0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 shutdown_connections 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf2006b1c0 0x7fdf2006d680 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdf4015bbe0 0x7fdf400ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf4015bfc0 0x7fdf400c8e70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 --2- 172.21.15.124:0/98148246 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdf400c63c0 0x7fdf40154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 >> 172.21.15.124:0/98148246 conn(0x7fdf400b2c10 msgr2=0x7fdf4006e7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 shutdown_connections 2023-12-13T23:55:16.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.365+0000 7fdf44cd3640 1 -- 172.21.15.124:0/98148246 wait complete. 2023-12-13T23:55:16.373 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-12-13T23:55:16.109984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2928640,"data_stored":1758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.367307"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741160+0000","last_change":"2023-12-13T23:55:12.749065+0000","last_active":"2023-12-13T23:55:13.741160+0000","last_peered":"2023-12-13T23:55:13.741160+0000","last_clean":"2023-12-13T23:55:13.741160+0000","last_became_active":"2023-12-13T23:55:12.748399+0000","last_became_peered":"2023-12-13T23:55:12.748399+0000","last_unstale":"2023-12-13T23:55:13.741160+0000","last_undegraded":"2023-12-13T23:55:13.741160+0000","last_fullsized":"2023-12-13T23:55:13.741160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613725+0000","last_change":"2023-12-13T23:55:12.751938+0000","last_active":"2023-12-13T23:55:14.613725+0000","last_peered":"2023-12-13T23:55:14.613725+0000","last_clean":"2023-12-13T23:55:14.613725+0000","last_became_active":"2023-12-13T23:55:12.751688+0000","last_became_peered":"2023-12-13T23:55:12.751688+0000","last_unstale":"2023-12-13T23:55:14.613725+0000","last_undegraded":"2023-12-13T23:55:14.613725+0000","last_fullsized":"2023-12-13T23:55:14.613725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624101+0000","last_change":"2023-12-13T23:55:12.757831+0000","last_active":"2023-12-13T23:55:14.624101+0000","last_peered":"2023-12-13T23:55:14.624101+0000","last_clean":"2023-12-13T23:55:14.624101+0000","last_became_active":"2023-12-13T23:55:12.757403+0000","last_became_peered":"2023-12-13T23:55:12.757403+0000","last_unstale":"2023-12-13T23:55:14.624101+0000","last_undegraded":"2023-12-13T23:55:14.624101+0000","last_fullsized":"2023-12-13T23:55:14.624101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624401+0000","last_change":"2023-12-13T23:55:12.757838+0000","last_active":"2023-12-13T23:55:14.624401+0000","last_peered":"2023-12-13T23:55:14.624401+0000","last_clean":"2023-12-13T23:55:14.624401+0000","last_became_active":"2023-12-13T23:55:12.757590+0000","last_became_peered":"2023-12-13T23:55:12.757590+0000","last_unstale":"2023-12-13T23:55:14.624401+0000","last_undegraded":"2023-12-13T23:55:14.624401+0000","last_fullsized":"2023-12-13T23:55:14.624401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740506+0000","last_change":"2023-12-13T23:55:12.750174+0000","last_active":"2023-12-13T23:55:13.740506+0000","last_peered":"2023-12-13T23:55:13.740506+0000","last_clean":"2023-12-13T23:55:13.740506+0000","last_became_active":"2023-12-13T23:55:12.749937+0000","last_became_peered":"2023-12-13T23:55:12.749937+0000","last_unstale":"2023-12-13T23:55:13.740506+0000","last_undegraded":"2023-12-13T23:55:13.740506+0000","last_fullsized":"2023-12-13T23:55:13.740506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624775+0000","last_change":"2023-12-13T23:55:12.760175+0000","last_active":"2023-12-13T23:55:14.624775+0000","last_peered":"2023-12-13T23:55:14.624775+0000","last_clean":"2023-12-13T23:55:14.624775+0000","last_became_active":"2023-12-13T23:55:12.759888+0000","last_became_peered":"2023-12-13T23:55:12.759888+0000","last_unstale":"2023-12-13T23:55:14.624775+0000","last_undegraded":"2023-12-13T23:55:14.624775+0000","last_fullsized":"2023-12-13T23:55:14.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740922+0000","last_change":"2023-12-13T23:55:12.758545+0000","last_active":"2023-12-13T23:55:13.740922+0000","last_peered":"2023-12-13T23:55:13.740922+0000","last_clean":"2023-12-13T23:55:13.740922+0000","last_became_active":"2023-12-13T23:55:12.758292+0000","last_became_peered":"2023-12-13T23:55:12.758292+0000","last_unstale":"2023-12-13T23:55:13.740922+0000","last_undegraded":"2023-12-13T23:55:13.740922+0000","last_fullsized":"2023-12-13T23:55:13.740922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737487+0000","last_change":"2023-12-13T23:55:12.756050+0000","last_active":"2023-12-13T23:55:13.737487+0000","last_peered":"2023-12-13T23:55:13.737487+0000","last_clean":"2023-12-13T23:55:13.737487+0000","last_became_active":"2023-12-13T23:55:12.755765+0000","last_became_peered":"2023-12-13T23:55:12.755765+0000","last_unstale":"2023-12-13T23:55:13.737487+0000","last_undegraded":"2023-12-13T23:55:13.737487+0000","last_fullsized":"2023-12-13T23:55:13.737487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741293+0000","last_change":"2023-12-13T23:55:12.751022+0000","last_active":"2023-12-13T23:55:13.741293+0000","last_peered":"2023-12-13T23:55:13.741293+0000","last_clean":"2023-12-13T23:55:13.741293+0000","last_became_active":"2023-12-13T23:55:12.750750+0000","last_became_peered":"2023-12-13T23:55:12.750750+0000","last_unstale":"2023-12-13T23:55:13.741293+0000","last_undegraded":"2023-12-13T23:55:13.741293+0000","last_fullsized":"2023-12-13T23:55:13.741293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624534+0000","last_change":"2023-12-13T23:55:12.754201+0000","last_active":"2023-12-13T23:55:14.624534+0000","last_peered":"2023-12-13T23:55:14.624534+0000","last_clean":"2023-12-13T23:55:14.624534+0000","last_became_active":"2023-12-13T23:55:12.753905+0000","last_became_peered":"2023-12-13T23:55:12.753905+0000","last_unstale":"2023-12-13T23:55:14.624534+0000","last_undegraded":"2023-12-13T23:55:14.624534+0000","last_fullsized":"2023-12-13T23:55:14.624534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624445+0000","last_change":"2023-12-13T23:55:12.756766+0000","last_active":"2023-12-13T23:55:14.624445+0000","last_peered":"2023-12-13T23:55:14.624445+0000","last_clean":"2023-12-13T23:55:14.624445+0000","last_became_active":"2023-12-13T23:55:12.756500+0000","last_became_peered":"2023-12-13T23:55:12.756500+0000","last_unstale":"2023-12-13T23:55:14.624445+0000","last_undegraded":"2023-12-13T23:55:14.624445+0000","last_fullsized":"2023-12-13T23:55:14.624445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741255+0000","last_change":"2023-12-13T23:55:12.756454+0000","last_active":"2023-12-13T23:55:13.741255+0000","last_peered":"2023-12-13T23:55:13.741255+0000","last_clean":"2023-12-13T23:55:13.741255+0000","last_became_active":"2023-12-13T23:55:12.756151+0000","last_became_peered":"2023-12-13T23:55:12.756151+0000","last_unstale":"2023-12-13T23:55:13.741255+0000","last_undegraded":"2023-12-13T23:55:13.741255+0000","last_fullsized":"2023-12-13T23:55:13.741255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:41.266746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737767+0000","last_change":"2023-12-13T23:55:12.751798+0000","last_active":"2023-12-13T23:55:13.737767+0000","last_peered":"2023-12-13T23:55:13.737767+0000","last_clean":"2023-12-13T23:55:13.737767+0000","last_became_active":"2023-12-13T23:55:12.751513+0000","last_became_peered":"2023-12-13T23:55:12.751513+0000","last_unstale":"2023-12-13T23:55:13.737767+0000","last_undegraded":"2023-12-13T23:55:13.737767+0000","last_fullsized":"2023-12-13T23:55:13.737767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624265+0000","last_change":"2023-12-13T23:55:12.757612+0000","last_active":"2023-12-13T23:55:14.624265+0000","last_peered":"2023-12-13T23:55:14.624265+0000","last_clean":"2023-12-13T23:55:14.624265+0000","last_became_active":"2023-12-13T23:55:12.757320+0000","last_became_peered":"2023-12-13T23:55:12.757320+0000","last_unstale":"2023-12-13T23:55:14.624265+0000","last_undegraded":"2023-12-13T23:55:14.624265+0000","last_fullsized":"2023-12-13T23:55:14.624265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613523+0000","last_change":"2023-12-13T23:55:12.743417+0000","last_active":"2023-12-13T23:55:14.613523+0000","last_peered":"2023-12-13T23:55:14.613523+0000","last_clean":"2023-12-13T23:55:14.613523+0000","last_became_active":"2023-12-13T23:55:12.743121+0000","last_became_peered":"2023-12-13T23:55:12.743121+0000","last_unstale":"2023-12-13T23:55:14.613523+0000","last_undegraded":"2023-12-13T23:55:14.613523+0000","last_fullsized":"2023-12-13T23:55:14.613523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624106+0000","last_change":"2023-12-13T23:55:12.761124+0000","last_active":"2023-12-13T23:55:14.624106+0000","last_peered":"2023-12-13T23:55:14.624106+0000","last_clean":"2023-12-13T23:55:14.624106+0000","last_became_active":"2023-12-13T23:55:12.760661+0000","last_became_peered":"2023-12-13T23:55:12.760661+0000","last_unstale":"2023-12-13T23:55:14.624106+0000","last_undegraded":"2023-12-13T23:55:14.624106+0000","last_fullsized":"2023-12-13T23:55:14.624106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624905+0000","last_change":"2023-12-13T23:55:12.752470+0000","last_active":"2023-12-13T23:55:14.624905+0000","last_peered":"2023-12-13T23:55:14.624905+0000","last_clean":"2023-12-13T23:55:14.624905+0000","last_became_active":"2023-12-13T23:55:12.752197+0000","last_became_peered":"2023-12-13T23:55:12.752197+0000","last_unstale":"2023-12-13T23:55:14.624905+0000","last_undegraded":"2023-12-13T23:55:14.624905+0000","last_fullsized":"2023-12-13T23:55:14.624905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623990+0000","last_change":"2023-12-13T23:55:12.748844+0000","last_active":"2023-12-13T23:55:14.623990+0000","last_peered":"2023-12-13T23:55:14.623990+0000","last_clean":"2023-12-13T23:55:14.623990+0000","last_became_active":"2023-12-13T23:55:12.748564+0000","last_became_peered":"2023-12-13T23:55:12.748564+0000","last_unstale":"2023-12-13T23:55:14.623990+0000","last_undegraded":"2023-12-13T23:55:14.623990+0000","last_fullsized":"2023-12-13T23:55:14.623990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741142+0000","last_change":"2023-12-13T23:55:12.749584+0000","last_active":"2023-12-13T23:55:13.741142+0000","last_peered":"2023-12-13T23:55:13.741142+0000","last_clean":"2023-12-13T23:55:13.741142+0000","last_became_active":"2023-12-13T23:55:12.749330+0000","last_became_peered":"2023-12-13T23:55:12.749330+0000","last_unstale":"2023-12-13T23:55:13.741142+0000","last_undegraded":"2023-12-13T23:55:13.741142+0000","last_fullsized":"2023-12-13T23:55:13.741142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624156+0000","last_change":"2023-12-13T23:55:12.746390+0000","last_active":"2023-12-13T23:55:14.624156+0000","last_peered":"2023-12-13T23:55:14.624156+0000","last_clean":"2023-12-13T23:55:14.624156+0000","last_became_active":"2023-12-13T23:55:12.746135+0000","last_became_peered":"2023-12-13T23:55:12.746135+0000","last_unstale":"2023-12-13T23:55:14.624156+0000","last_undegraded":"2023-12-13T23:55:14.624156+0000","last_fullsized":"2023-12-13T23:55:14.624156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624034+0000","last_change":"2023-12-13T23:55:12.752805+0000","last_active":"2023-12-13T23:55:14.624034+0000","last_peered":"2023-12-13T23:55:14.624034+0000","last_clean":"2023-12-13T23:55:14.624034+0000","last_became_active":"2023-12-13T23:55:12.752447+0000","last_became_peered":"2023-12-13T23:55:12.752447+0000","last_unstale":"2023-12-13T23:55:14.624034+0000","last_undegraded":"2023-12-13T23:55:14.624034+0000","last_fullsized":"2023-12-13T23:55:14.624034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:24.291337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624917+0000","last_change":"2023-12-13T23:55:12.754499+0000","last_active":"2023-12-13T23:55:14.624917+0000","last_peered":"2023-12-13T23:55:14.624917+0000","last_clean":"2023-12-13T23:55:14.624917+0000","last_became_active":"2023-12-13T23:55:12.753996+0000","last_became_peered":"2023-12-13T23:55:12.753996+0000","last_unstale":"2023-12-13T23:55:14.624917+0000","last_undegraded":"2023-12-13T23:55:14.624917+0000","last_fullsized":"2023-12-13T23:55:14.624917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737534+0000","last_change":"2023-12-13T23:55:12.750598+0000","last_active":"2023-12-13T23:55:13.737534+0000","last_peered":"2023-12-13T23:55:13.737534+0000","last_clean":"2023-12-13T23:55:13.737534+0000","last_became_active":"2023-12-13T23:55:12.750296+0000","last_became_peered":"2023-12-13T23:55:12.750296+0000","last_unstale":"2023-12-13T23:55:13.737534+0000","last_undegraded":"2023-12-13T23:55:13.737534+0000","last_fullsized":"2023-12-13T23:55:13.737534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624327+0000","last_change":"2023-12-13T23:55:12.750435+0000","last_active":"2023-12-13T23:55:14.624327+0000","last_peered":"2023-12-13T23:55:14.624327+0000","last_clean":"2023-12-13T23:55:14.624327+0000","last_became_active":"2023-12-13T23:55:12.750151+0000","last_became_peered":"2023-12-13T23:55:12.750151+0000","last_unstale":"2023-12-13T23:55:14.624327+0000","last_undegraded":"2023-12-13T23:55:14.624327+0000","last_fullsized":"2023-12-13T23:55:14.624327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737900+0000","last_change":"2023-12-13T23:55:12.750495+0000","last_active":"2023-12-13T23:55:13.737900+0000","last_peered":"2023-12-13T23:55:13.737900+0000","last_clean":"2023-12-13T23:55:13.737900+0000","last_became_active":"2023-12-13T23:55:12.750227+0000","last_became_peered":"2023-12-13T23:55:12.750227+0000","last_unstale":"2023-12-13T23:55:13.737900+0000","last_undegraded":"2023-12-13T23:55:13.737900+0000","last_fullsized":"2023-12-13T23:55:13.737900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624667+0000","last_change":"2023-12-13T23:55:12.751349+0000","last_active":"2023-12-13T23:55:14.624667+0000","last_peered":"2023-12-13T23:55:14.624667+0000","last_clean":"2023-12-13T23:55:14.624667+0000","last_became_active":"2023-12-13T23:55:12.751057+0000","last_became_peered":"2023-12-13T23:55:12.751057+0000","last_unstale":"2023-12-13T23:55:14.624667+0000","last_undegraded":"2023-12-13T23:55:14.624667+0000","last_fullsized":"2023-12-13T23:55:14.624667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613647+0000","last_change":"2023-12-13T23:55:12.748079+0000","last_active":"2023-12-13T23:55:14.613647+0000","last_peered":"2023-12-13T23:55:14.613647+0000","last_clean":"2023-12-13T23:55:14.613647+0000","last_became_active":"2023-12-13T23:55:12.747722+0000","last_became_peered":"2023-12-13T23:55:12.747722+0000","last_unstale":"2023-12-13T23:55:14.613647+0000","last_undegraded":"2023-12-13T23:55:14.613647+0000","last_fullsized":"2023-12-13T23:55:14.613647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737802+0000","last_change":"2023-12-13T23:55:12.747671+0000","last_active":"2023-12-13T23:55:13.737802+0000","last_peered":"2023-12-13T23:55:13.737802+0000","last_clean":"2023-12-13T23:55:13.737802+0000","last_became_active":"2023-12-13T23:55:12.747431+0000","last_became_peered":"2023-12-13T23:55:12.747431+0000","last_unstale":"2023-12-13T23:55:13.737802+0000","last_undegraded":"2023-12-13T23:55:13.737802+0000","last_fullsized":"2023-12-13T23:55:13.737802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624186+0000","last_change":"2023-12-13T23:55:12.751994+0000","last_active":"2023-12-13T23:55:14.624186+0000","last_peered":"2023-12-13T23:55:14.624186+0000","last_clean":"2023-12-13T23:55:14.624186+0000","last_became_active":"2023-12-13T23:55:12.751598+0000","last_became_peered":"2023-12-13T23:55:12.751598+0000","last_unstale":"2023-12-13T23:55:14.624186+0000","last_undegraded":"2023-12-13T23:55:14.624186+0000","last_fullsized":"2023-12-13T23:55:14.624186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624938+0000","last_change":"2023-12-13T23:55:12.754312+0000","last_active":"2023-12-13T23:55:14.624938+0000","last_peered":"2023-12-13T23:55:14.624938+0000","last_clean":"2023-12-13T23:55:14.624938+0000","last_became_active":"2023-12-13T23:55:12.753618+0000","last_became_peered":"2023-12-13T23:55:12.753618+0000","last_unstale":"2023-12-13T23:55:14.624938+0000","last_undegraded":"2023-12-13T23:55:14.624938+0000","last_fullsized":"2023-12-13T23:55:14.624938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623822+0000","last_change":"2023-12-13T23:55:12.755004+0000","last_active":"2023-12-13T23:55:14.623822+0000","last_peered":"2023-12-13T23:55:14.623822+0000","last_clean":"2023-12-13T23:55:14.623822+0000","last_became_active":"2023-12-13T23:55:12.754761+0000","last_became_peered":"2023-12-13T23:55:12.754761+0000","last_unstale":"2023-12-13T23:55:14.623822+0000","last_undegraded":"2023-12-13T23:55:14.623822+0000","last_fullsized":"2023-12-13T23:55:14.623822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624213+0000","last_change":"2023-12-13T23:55:12.758828+0000","last_active":"2023-12-13T23:55:14.624213+0000","last_peered":"2023-12-13T23:55:14.624213+0000","last_clean":"2023-12-13T23:55:14.624213+0000","last_became_active":"2023-12-13T23:55:12.758520+0000","last_became_peered":"2023-12-13T23:55:12.758520+0000","last_unstale":"2023-12-13T23:55:14.624213+0000","last_undegraded":"2023-12-13T23:55:14.624213+0000","last_fullsized":"2023-12-13T23:55:14.624213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740370+0000","last_change":"2023-12-13T23:55:12.745882+0000","last_active":"2023-12-13T23:55:13.740370+0000","last_peered":"2023-12-13T23:55:13.740370+0000","last_clean":"2023-12-13T23:55:13.740370+0000","last_became_active":"2023-12-13T23:55:12.744988+0000","last_became_peered":"2023-12-13T23:55:12.744988+0000","last_unstale":"2023-12-13T23:55:13.740370+0000","last_undegraded":"2023-12-13T23:55:13.740370+0000","last_fullsized":"2023-12-13T23:55:13.740370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:57.837875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737851+0000","last_change":"2023-12-13T23:55:11.769713+0000","last_active":"2023-12-13T23:55:13.737851+0000","last_peered":"2023-12-13T23:55:13.737851+0000","last_clean":"2023-12-13T23:55:13.737851+0000","last_became_active":"2023-12-13T23:55:11.769540+0000","last_became_peered":"2023-12-13T23:55:11.769540+0000","last_unstale":"2023-12-13T23:55:13.737851+0000","last_undegraded":"2023-12-13T23:55:13.737851+0000","last_fullsized":"2023-12-13T23:55:13.737851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624177+0000","last_change":"2023-12-13T23:55:12.752916+0000","last_active":"2023-12-13T23:55:14.624177+0000","last_peered":"2023-12-13T23:55:14.624177+0000","last_clean":"2023-12-13T23:55:14.624177+0000","last_became_active":"2023-12-13T23:55:12.751877+0000","last_became_peered":"2023-12-13T23:55:12.751877+0000","last_unstale":"2023-12-13T23:55:14.624177+0000","last_undegraded":"2023-12-13T23:55:14.624177+0000","last_fullsized":"2023-12-13T23:55:14.624177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624064+0000","last_change":"2023-12-13T23:55:11.767168+0000","last_active":"2023-12-13T23:55:14.624064+0000","last_peered":"2023-12-13T23:55:14.624064+0000","last_clean":"2023-12-13T23:55:14.624064+0000","last_became_active":"2023-12-13T23:55:11.766944+0000","last_became_peered":"2023-12-13T23:55:11.766944+0000","last_unstale":"2023-12-13T23:55:14.624064+0000","last_undegraded":"2023-12-13T23:55:14.624064+0000","last_fullsized":"2023-12-13T23:55:14.624064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737587+0000","last_change":"2023-12-13T23:55:11.764302+0000","last_active":"2023-12-13T23:55:13.737587+0000","last_peered":"2023-12-13T23:55:13.737587+0000","last_clean":"2023-12-13T23:55:13.737587+0000","last_became_active":"2023-12-13T23:55:11.763930+0000","last_became_peered":"2023-12-13T23:55:11.763930+0000","last_unstale":"2023-12-13T23:55:13.737587+0000","last_undegraded":"2023-12-13T23:55:13.737587+0000","last_fullsized":"2023-12-13T23:55:13.737587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737560+0000","last_change":"2023-12-13T23:55:12.760219+0000","last_active":"2023-12-13T23:55:13.737560+0000","last_peered":"2023-12-13T23:55:13.737560+0000","last_clean":"2023-12-13T23:55:13.737560+0000","last_became_active":"2023-12-13T23:55:12.759890+0000","last_became_peered":"2023-12-13T23:55:12.759890+0000","last_unstale":"2023-12-13T23:55:13.737560+0000","last_undegraded":"2023-12-13T23:55:13.737560+0000","last_fullsized":"2023-12-13T23:55:13.737560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624404+0000","last_change":"2023-12-13T23:55:11.763128+0000","last_active":"2023-12-13T23:55:14.624404+0000","last_peered":"2023-12-13T23:55:14.624404+0000","last_clean":"2023-12-13T23:55:14.624404+0000","last_became_active":"2023-12-13T23:55:11.762848+0000","last_became_peered":"2023-12-13T23:55:11.762848+0000","last_unstale":"2023-12-13T23:55:14.624404+0000","last_undegraded":"2023-12-13T23:55:14.624404+0000","last_fullsized":"2023-12-13T23:55:14.624404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613510+0000","last_change":"2023-12-13T23:55:12.747367+0000","last_active":"2023-12-13T23:55:14.613510+0000","last_peered":"2023-12-13T23:55:14.613510+0000","last_clean":"2023-12-13T23:55:14.613510+0000","last_became_active":"2023-12-13T23:55:12.747074+0000","last_became_peered":"2023-12-13T23:55:12.747074+0000","last_unstale":"2023-12-13T23:55:14.613510+0000","last_undegraded":"2023-12-13T23:55:14.613510+0000","last_fullsized":"2023-12-13T23:55:14.613510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624348+0000","last_change":"2023-12-13T23:55:11.766140+0000","last_active":"2023-12-13T23:55:14.624348+0000","last_peered":"2023-12-13T23:55:14.624348+0000","last_clean":"2023-12-13T23:55:14.624348+0000","last_became_active":"2023-12-13T23:55:11.765848+0000","last_became_peered":"2023-12-13T23:55:11.765848+0000","last_unstale":"2023-12-13T23:55:14.624348+0000","last_undegraded":"2023-12-13T23:55:14.624348+0000","last_fullsized":"2023-12-13T23:55:14.624348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624786+0000","last_change":"2023-12-13T23:55:12.750678+0000","last_active":"2023-12-13T23:55:14.624786+0000","last_peered":"2023-12-13T23:55:14.624786+0000","last_clean":"2023-12-13T23:55:14.624786+0000","last_became_active":"2023-12-13T23:55:12.750200+0000","last_became_peered":"2023-12-13T23:55:12.750200+0000","last_unstale":"2023-12-13T23:55:14.624786+0000","last_undegraded":"2023-12-13T23:55:14.624786+0000","last_fullsized":"2023-12-13T23:55:14.624786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624863+0000","last_change":"2023-12-13T23:55:11.763844+0000","last_active":"2023-12-13T23:55:14.624863+0000","last_peered":"2023-12-13T23:55:14.624863+0000","last_clean":"2023-12-13T23:55:14.624863+0000","last_became_active":"2023-12-13T23:55:11.763574+0000","last_became_peered":"2023-12-13T23:55:11.763574+0000","last_unstale":"2023-12-13T23:55:14.624863+0000","last_undegraded":"2023-12-13T23:55:14.624863+0000","last_fullsized":"2023-12-13T23:55:14.624863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:41:34.696890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624340+0000","last_change":"2023-12-13T23:55:12.754675+0000","last_active":"2023-12-13T23:55:14.624340+0000","last_peered":"2023-12-13T23:55:14.624340+0000","last_clean":"2023-12-13T23:55:14.624340+0000","last_became_active":"2023-12-13T23:55:12.753740+0000","last_became_peered":"2023-12-13T23:55:12.753740+0000","last_unstale":"2023-12-13T23:55:14.624340+0000","last_undegraded":"2023-12-13T23:55:14.624340+0000","last_fullsized":"2023-12-13T23:55:14.624340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613654+0000","last_change":"2023-12-13T23:55:11.765823+0000","last_active":"2023-12-13T23:55:14.613654+0000","last_peered":"2023-12-13T23:55:14.613654+0000","last_clean":"2023-12-13T23:55:14.613654+0000","last_became_active":"2023-12-13T23:55:11.765515+0000","last_became_peered":"2023-12-13T23:55:11.765515+0000","last_unstale":"2023-12-13T23:55:14.613654+0000","last_undegraded":"2023-12-13T23:55:14.613654+0000","last_fullsized":"2023-12-13T23:55:14.613654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741390+0000","last_change":"2023-12-13T23:55:12.750387+0000","last_active":"2023-12-13T23:55:13.741390+0000","last_peered":"2023-12-13T23:55:13.741390+0000","last_clean":"2023-12-13T23:55:13.741390+0000","last_became_active":"2023-12-13T23:55:12.749101+0000","last_became_peered":"2023-12-13T23:55:12.749101+0000","last_unstale":"2023-12-13T23:55:13.741390+0000","last_undegraded":"2023-12-13T23:55:13.741390+0000","last_fullsized":"2023-12-13T23:55:13.741390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624137+0000","last_change":"2023-12-13T23:55:11.763071+0000","last_active":"2023-12-13T23:55:14.624137+0000","last_peered":"2023-12-13T23:55:14.624137+0000","last_clean":"2023-12-13T23:55:14.624137+0000","last_became_active":"2023-12-13T23:55:11.762774+0000","last_became_peered":"2023-12-13T23:55:11.762774+0000","last_unstale":"2023-12-13T23:55:14.624137+0000","last_undegraded":"2023-12-13T23:55:14.624137+0000","last_fullsized":"2023-12-13T23:55:14.624137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740649+0000","last_change":"2023-12-13T23:55:12.747891+0000","last_active":"2023-12-13T23:55:13.740649+0000","last_peered":"2023-12-13T23:55:13.740649+0000","last_clean":"2023-12-13T23:55:13.740649+0000","last_became_active":"2023-12-13T23:55:12.747629+0000","last_became_peered":"2023-12-13T23:55:12.747629+0000","last_unstale":"2023-12-13T23:55:13.740649+0000","last_undegraded":"2023-12-13T23:55:13.740649+0000","last_fullsized":"2023-12-13T23:55:13.740649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624263+0000","last_change":"2023-12-13T23:55:11.768277+0000","last_active":"2023-12-13T23:55:14.624263+0000","last_peered":"2023-12-13T23:55:14.624263+0000","last_clean":"2023-12-13T23:55:14.624263+0000","last_became_active":"2023-12-13T23:55:11.768039+0000","last_became_peered":"2023-12-13T23:55:11.768039+0000","last_unstale":"2023-12-13T23:55:14.624263+0000","last_undegraded":"2023-12-13T23:55:14.624263+0000","last_fullsized":"2023-12-13T23:55:14.624263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624351+0000","last_change":"2023-12-13T23:55:12.758134+0000","last_active":"2023-12-13T23:55:14.624351+0000","last_peered":"2023-12-13T23:55:14.624351+0000","last_clean":"2023-12-13T23:55:14.624351+0000","last_became_active":"2023-12-13T23:55:12.757679+0000","last_became_peered":"2023-12-13T23:55:12.757679+0000","last_unstale":"2023-12-13T23:55:14.624351+0000","last_undegraded":"2023-12-13T23:55:14.624351+0000","last_fullsized":"2023-12-13T23:55:14.624351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625036+0000","last_change":"2023-12-13T23:55:11.765748+0000","last_active":"2023-12-13T23:55:14.625036+0000","last_peered":"2023-12-13T23:55:14.625036+0000","last_clean":"2023-12-13T23:55:14.625036+0000","last_became_active":"2023-12-13T23:55:11.765320+0000","last_became_peered":"2023-12-13T23:55:11.765320+0000","last_unstale":"2023-12-13T23:55:14.625036+0000","last_undegraded":"2023-12-13T23:55:14.625036+0000","last_fullsized":"2023-12-13T23:55:14.625036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740530+0000","last_change":"2023-12-13T23:55:12.747127+0000","last_active":"2023-12-13T23:55:13.740530+0000","last_peered":"2023-12-13T23:55:13.740530+0000","last_clean":"2023-12-13T23:55:13.740530+0000","last_became_active":"2023-12-13T23:55:12.746838+0000","last_became_peered":"2023-12-13T23:55:12.746838+0000","last_unstale":"2023-12-13T23:55:13.740530+0000","last_undegraded":"2023-12-13T23:55:13.740530+0000","last_fullsized":"2023-12-13T23:55:13.740530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623961+0000","last_change":"2023-12-13T23:55:11.765098+0000","last_active":"2023-12-13T23:55:14.623961+0000","last_peered":"2023-12-13T23:55:14.623961+0000","last_clean":"2023-12-13T23:55:14.623961+0000","last_became_active":"2023-12-13T23:55:11.764746+0000","last_became_peered":"2023-12-13T23:55:11.764746+0000","last_unstale":"2023-12-13T23:55:14.623961+0000","last_undegraded":"2023-12-13T23:55:14.623961+0000","last_fullsized":"2023-12-13T23:55:14.623961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624195+0000","last_change":"2023-12-13T23:55:12.759703+0000","last_active":"2023-12-13T23:55:14.624195+0000","last_peered":"2023-12-13T23:55:14.624195+0000","last_clean":"2023-12-13T23:55:14.624195+0000","last_became_active":"2023-12-13T23:55:12.759371+0000","last_became_peered":"2023-12-13T23:55:12.759371+0000","last_unstale":"2023-12-13T23:55:14.624195+0000","last_undegraded":"2023-12-13T23:55:14.624195+0000","last_fullsized":"2023-12-13T23:55:14.624195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623693+0000","last_change":"2023-12-13T23:55:11.757065+0000","last_active":"2023-12-13T23:55:14.623693+0000","last_peered":"2023-12-13T23:55:14.623693+0000","last_clean":"2023-12-13T23:55:14.623693+0000","last_became_active":"2023-12-13T23:55:11.756732+0000","last_became_peered":"2023-12-13T23:55:11.756732+0000","last_unstale":"2023-12-13T23:55:14.623693+0000","last_undegraded":"2023-12-13T23:55:14.623693+0000","last_fullsized":"2023-12-13T23:55:14.623693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740938+0000","last_change":"2023-12-13T23:55:12.747168+0000","last_active":"2023-12-13T23:55:13.740938+0000","last_peered":"2023-12-13T23:55:13.740938+0000","last_clean":"2023-12-13T23:55:13.740938+0000","last_became_active":"2023-12-13T23:55:12.746860+0000","last_became_peered":"2023-12-13T23:55:12.746860+0000","last_unstale":"2023-12-13T23:55:13.740938+0000","last_undegraded":"2023-12-13T23:55:13.740938+0000","last_fullsized":"2023-12-13T23:55:13.740938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740851+0000","last_change":"2023-12-13T23:55:11.765922+0000","last_active":"2023-12-13T23:55:13.740851+0000","last_peered":"2023-12-13T23:55:13.740851+0000","last_clean":"2023-12-13T23:55:13.740851+0000","last_became_active":"2023-12-13T23:55:11.765641+0000","last_became_peered":"2023-12-13T23:55:11.765641+0000","last_unstale":"2023-12-13T23:55:13.740851+0000","last_undegraded":"2023-12-13T23:55:13.740851+0000","last_fullsized":"2023-12-13T23:55:13.740851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:07:21.569553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741605+0000","last_change":"2023-12-13T23:55:12.747433+0000","last_active":"2023-12-13T23:55:13.741605+0000","last_peered":"2023-12-13T23:55:13.741605+0000","last_clean":"2023-12-13T23:55:13.741605+0000","last_became_active":"2023-12-13T23:55:12.747169+0000","last_became_peered":"2023-12-13T23:55:12.747169+0000","last_unstale":"2023-12-13T23:55:13.741605+0000","last_undegraded":"2023-12-13T23:55:13.741605+0000","last_fullsized":"2023-12-13T23:55:13.741605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:03:57.776819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624072+0000","last_change":"2023-12-13T23:55:11.762985+0000","last_active":"2023-12-13T23:55:14.624072+0000","last_peered":"2023-12-13T23:55:14.624072+0000","last_clean":"2023-12-13T23:55:14.624072+0000","last_became_active":"2023-12-13T23:55:11.762654+0000","last_became_peered":"2023-12-13T23:55:11.762654+0000","last_unstale":"2023-12-13T23:55:14.624072+0000","last_undegraded":"2023-12-13T23:55:14.624072+0000","last_fullsized":"2023-12-13T23:55:14.624072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741322+0000","last_change":"2023-12-13T23:55:12.756265+0000","last_active":"2023-12-13T23:55:13.741322+0000","last_peered":"2023-12-13T23:55:13.741322+0000","last_clean":"2023-12-13T23:55:13.741322+0000","last_became_active":"2023-12-13T23:55:12.756003+0000","last_became_peered":"2023-12-13T23:55:12.756003+0000","last_unstale":"2023-12-13T23:55:13.741322+0000","last_undegraded":"2023-12-13T23:55:13.741322+0000","last_fullsized":"2023-12-13T23:55:13.741322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:26:45.885932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613709+0000","last_change":"2023-12-13T23:55:11.766945+0000","last_active":"2023-12-13T23:55:14.613709+0000","last_peered":"2023-12-13T23:55:14.613709+0000","last_clean":"2023-12-13T23:55:14.613709+0000","last_became_active":"2023-12-13T23:55:11.766671+0000","last_became_peered":"2023-12-13T23:55:11.766671+0000","last_unstale":"2023-12-13T23:55:14.613709+0000","last_undegraded":"2023-12-13T23:55:14.613709+0000","last_fullsized":"2023-12-13T23:55:14.613709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624646+0000","last_change":"2023-12-13T23:55:12.752034+0000","last_active":"2023-12-13T23:55:14.624646+0000","last_peered":"2023-12-13T23:55:14.624646+0000","last_clean":"2023-12-13T23:55:14.624646+0000","last_became_active":"2023-12-13T23:55:12.751764+0000","last_became_peered":"2023-12-13T23:55:12.751764+0000","last_unstale":"2023-12-13T23:55:14.624646+0000","last_undegraded":"2023-12-13T23:55:14.624646+0000","last_fullsized":"2023-12-13T23:55:14.624646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623912+0000","last_change":"2023-12-13T23:55:11.765879+0000","last_active":"2023-12-13T23:55:14.623912+0000","last_peered":"2023-12-13T23:55:14.623912+0000","last_clean":"2023-12-13T23:55:14.623912+0000","last_became_active":"2023-12-13T23:55:11.765607+0000","last_became_peered":"2023-12-13T23:55:11.765607+0000","last_unstale":"2023-12-13T23:55:14.623912+0000","last_undegraded":"2023-12-13T23:55:14.623912+0000","last_fullsized":"2023-12-13T23:55:14.623912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:40.262294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740784+0000","last_change":"2023-12-13T23:55:12.756349+0000","last_active":"2023-12-13T23:55:13.740784+0000","last_peered":"2023-12-13T23:55:13.740784+0000","last_clean":"2023-12-13T23:55:13.740784+0000","last_became_active":"2023-12-13T23:55:12.756033+0000","last_became_peered":"2023-12-13T23:55:12.756033+0000","last_unstale":"2023-12-13T23:55:13.740784+0000","last_undegraded":"2023-12-13T23:55:13.740784+0000","last_fullsized":"2023-12-13T23:55:13.740784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-13T23:55:12.739242+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:12.739242+0000","last_peered":"2023-12-13T23:55:12.739242+0000","last_clean":"2023-12-13T23:55:12.739242+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:12.739242+0000","last_undegraded":"2023-12-13T23:55:12.739242+0000","last_fullsized":"2023-12-13T23:55:12.739242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624508+0000","last_change":"2023-12-13T23:55:11.758971+0000","last_active":"2023-12-13T23:55:14.624508+0000","last_peered":"2023-12-13T23:55:14.624508+0000","last_clean":"2023-12-13T23:55:14.624508+0000","last_became_active":"2023-12-13T23:55:11.758594+0000","last_became_peered":"2023-12-13T23:55:11.758594+0000","last_unstale":"2023-12-13T23:55:14.624508+0000","last_undegraded":"2023-12-13T23:55:14.624508+0000","last_fullsized":"2023-12-13T23:55:14.624508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737699+0000","last_change":"2023-12-13T23:55:12.755959+0000","last_active":"2023-12-13T23:55:13.737699+0000","last_peered":"2023-12-13T23:55:13.737699+0000","last_clean":"2023-12-13T23:55:13.737699+0000","last_became_active":"2023-12-13T23:55:12.755662+0000","last_became_peered":"2023-12-13T23:55:12.755662+0000","last_unstale":"2023-12-13T23:55:13.737699+0000","last_undegraded":"2023-12-13T23:55:13.737699+0000","last_fullsized":"2023-12-13T23:55:13.737699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740316+0000","last_change":"2023-12-13T23:55:11.767937+0000","last_active":"2023-12-13T23:55:13.740316+0000","last_peered":"2023-12-13T23:55:13.740316+0000","last_clean":"2023-12-13T23:55:13.740316+0000","last_became_active":"2023-12-13T23:55:11.767613+0000","last_became_peered":"2023-12-13T23:55:11.767613+0000","last_unstale":"2023-12-13T23:55:13.740316+0000","last_undegraded":"2023-12-13T23:55:13.740316+0000","last_fullsized":"2023-12-13T23:55:13.740316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:53.188642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624112+0000","last_change":"2023-12-13T23:55:12.757755+0000","last_active":"2023-12-13T23:55:14.624112+0000","last_peered":"2023-12-13T23:55:14.624112+0000","last_clean":"2023-12-13T23:55:14.624112+0000","last_became_active":"2023-12-13T23:55:12.757456+0000","last_became_peered":"2023-12-13T23:55:12.757456+0000","last_unstale":"2023-12-13T23:55:14.624112+0000","last_undegraded":"2023-12-13T23:55:14.624112+0000","last_fullsized":"2023-12-13T23:55:14.624112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623940+0000","last_change":"2023-12-13T23:55:11.755829+0000","last_active":"2023-12-13T23:55:14.623940+0000","last_peered":"2023-12-13T23:55:14.623940+0000","last_clean":"2023-12-13T23:55:14.623940+0000","last_became_active":"2023-12-13T23:55:11.755573+0000","last_became_peered":"2023-12-13T23:55:11.755573+0000","last_unstale":"2023-12-13T23:55:14.623940+0000","last_undegraded":"2023-12-13T23:55:14.623940+0000","last_fullsized":"2023-12-13T23:55:14.623940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:56.833424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613504+0000","last_change":"2023-12-13T23:55:12.749732+0000","last_active":"2023-12-13T23:55:14.613504+0000","last_peered":"2023-12-13T23:55:14.613504+0000","last_clean":"2023-12-13T23:55:14.613504+0000","last_became_active":"2023-12-13T23:55:12.749473+0000","last_became_peered":"2023-12-13T23:55:12.749473+0000","last_unstale":"2023-12-13T23:55:14.613504+0000","last_undegraded":"2023-12-13T23:55:14.613504+0000","last_fullsized":"2023-12-13T23:55:14.613504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625041+0000","last_change":"2023-12-13T23:55:11.765970+0000","last_active":"2023-12-13T23:55:14.625041+0000","last_peered":"2023-12-13T23:55:14.625041+0000","last_clean":"2023-12-13T23:55:14.625041+0000","last_became_active":"2023-12-13T23:55:11.765629+0000","last_became_peered":"2023-12-13T23:55:11.765629+0000","last_unstale":"2023-12-13T23:55:14.625041+0000","last_undegraded":"2023-12-13T23:55:14.625041+0000","last_fullsized":"2023-12-13T23:55:14.625041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741379+0000","last_change":"2023-12-13T23:55:12.758873+0000","last_active":"2023-12-13T23:55:13.741379+0000","last_peered":"2023-12-13T23:55:13.741379+0000","last_clean":"2023-12-13T23:55:13.741379+0000","last_became_active":"2023-12-13T23:55:12.758618+0000","last_became_peered":"2023-12-13T23:55:12.758618+0000","last_unstale":"2023-12-13T23:55:13.741379+0000","last_undegraded":"2023-12-13T23:55:13.741379+0000","last_fullsized":"2023-12-13T23:55:13.741379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613597+0000","last_change":"2023-12-13T23:55:11.762944+0000","last_active":"2023-12-13T23:55:14.613597+0000","last_peered":"2023-12-13T23:55:14.613597+0000","last_clean":"2023-12-13T23:55:14.613597+0000","last_became_active":"2023-12-13T23:55:11.762691+0000","last_became_peered":"2023-12-13T23:55:11.762691+0000","last_unstale":"2023-12-13T23:55:14.613597+0000","last_undegraded":"2023-12-13T23:55:14.613597+0000","last_fullsized":"2023-12-13T23:55:14.613597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625017+0000","last_change":"2023-12-13T23:55:12.755585+0000","last_active":"2023-12-13T23:55:14.625017+0000","last_peered":"2023-12-13T23:55:14.625017+0000","last_clean":"2023-12-13T23:55:14.625017+0000","last_became_active":"2023-12-13T23:55:12.755284+0000","last_became_peered":"2023-12-13T23:55:12.755284+0000","last_unstale":"2023-12-13T23:55:14.625017+0000","last_undegraded":"2023-12-13T23:55:14.625017+0000","last_fullsized":"2023-12-13T23:55:14.625017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741015+0000","last_change":"2023-12-13T23:55:11.770291+0000","last_active":"2023-12-13T23:55:13.741015+0000","last_peered":"2023-12-13T23:55:13.741015+0000","last_clean":"2023-12-13T23:55:13.741015+0000","last_became_active":"2023-12-13T23:55:11.769672+0000","last_became_peered":"2023-12-13T23:55:11.769672+0000","last_unstale":"2023-12-13T23:55:13.741015+0000","last_undegraded":"2023-12-13T23:55:13.741015+0000","last_fullsized":"2023-12-13T23:55:13.741015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613615+0000","last_change":"2023-12-13T23:55:12.749467+0000","last_active":"2023-12-13T23:55:14.613615+0000","last_peered":"2023-12-13T23:55:14.613615+0000","last_clean":"2023-12-13T23:55:14.613615+0000","last_became_active":"2023-12-13T23:55:12.749185+0000","last_became_peered":"2023-12-13T23:55:12.749185+0000","last_unstale":"2023-12-13T23:55:14.613615+0000","last_undegraded":"2023-12-13T23:55:14.613615+0000","last_fullsized":"2023-12-13T23:55:14.613615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613740+0000","last_change":"2023-12-13T23:55:11.763643+0000","last_active":"2023-12-13T23:55:14.613740+0000","last_peered":"2023-12-13T23:55:14.613740+0000","last_clean":"2023-12-13T23:55:14.613740+0000","last_became_active":"2023-12-13T23:55:11.763378+0000","last_became_peered":"2023-12-13T23:55:11.763378+0000","last_unstale":"2023-12-13T23:55:14.613740+0000","last_undegraded":"2023-12-13T23:55:14.613740+0000","last_fullsized":"2023-12-13T23:55:14.613740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737984+0000","last_change":"2023-12-13T23:55:12.752448+0000","last_active":"2023-12-13T23:55:13.737984+0000","last_peered":"2023-12-13T23:55:13.737984+0000","last_clean":"2023-12-13T23:55:13.737984+0000","last_became_active":"2023-12-13T23:55:12.752155+0000","last_became_peered":"2023-12-13T23:55:12.752155+0000","last_unstale":"2023-12-13T23:55:13.737984+0000","last_undegraded":"2023-12-13T23:55:13.737984+0000","last_fullsized":"2023-12-13T23:55:13.737984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624790+0000","last_change":"2023-12-13T23:55:11.768156+0000","last_active":"2023-12-13T23:55:14.624790+0000","last_peered":"2023-12-13T23:55:14.624790+0000","last_clean":"2023-12-13T23:55:14.624790+0000","last_became_active":"2023-12-13T23:55:11.767878+0000","last_became_peered":"2023-12-13T23:55:11.767878+0000","last_unstale":"2023-12-13T23:55:14.624790+0000","last_undegraded":"2023-12-13T23:55:14.624790+0000","last_fullsized":"2023-12-13T23:55:14.624790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624226+0000","last_change":"2023-12-13T23:55:12.751906+0000","last_active":"2023-12-13T23:55:14.624226+0000","last_peered":"2023-12-13T23:55:14.624226+0000","last_clean":"2023-12-13T23:55:14.624226+0000","last_became_active":"2023-12-13T23:55:12.751468+0000","last_became_peered":"2023-12-13T23:55:12.751468+0000","last_unstale":"2023-12-13T23:55:14.624226+0000","last_undegraded":"2023-12-13T23:55:14.624226+0000","last_fullsized":"2023-12-13T23:55:14.624226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740225+0000","last_change":"2023-12-13T23:55:11.768417+0000","last_active":"2023-12-13T23:55:13.740225+0000","last_peered":"2023-12-13T23:55:13.740225+0000","last_clean":"2023-12-13T23:55:13.740225+0000","last_became_active":"2023-12-13T23:55:11.768141+0000","last_became_peered":"2023-12-13T23:55:11.768141+0000","last_unstale":"2023-12-13T23:55:13.740225+0000","last_undegraded":"2023-12-13T23:55:13.740225+0000","last_fullsized":"2023-12-13T23:55:13.740225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740516+0000","last_change":"2023-12-13T23:55:12.751650+0000","last_active":"2023-12-13T23:55:13.740516+0000","last_peered":"2023-12-13T23:55:13.740516+0000","last_clean":"2023-12-13T23:55:13.740516+0000","last_became_active":"2023-12-13T23:55:12.749884+0000","last_became_peered":"2023-12-13T23:55:12.749884+0000","last_unstale":"2023-12-13T23:55:13.740516+0000","last_undegraded":"2023-12-13T23:55:13.740516+0000","last_fullsized":"2023-12-13T23:55:13.740516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:12.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613607+0000","last_change":"2023-12-13T23:55:11.761362+0000","last_active":"2023-12-13T23:55:14.613607+0000","last_peered":"2023-12-13T23:55:14.613607+0000","last_clean":"2023-12-13T23:55:14.613607+0000","last_became_active":"2023-12-13T23:55:11.761091+0000","last_became_peered":"2023-12-13T23:55:11.761091+0000","last_unstale":"2023-12-13T23:55:14.613607+0000","last_undegraded":"2023-12-13T23:55:14.613607+0000","last_fullsized":"2023-12-13T23:55:14.613607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:45.292370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737621+0000","last_change":"2023-12-13T23:55:12.747019+0000","last_active":"2023-12-13T23:55:13.737621+0000","last_peered":"2023-12-13T23:55:13.737621+0000","last_clean":"2023-12-13T23:55:13.737621+0000","last_became_active":"2023-12-13T23:55:12.746720+0000","last_became_peered":"2023-12-13T23:55:12.746720+0000","last_unstale":"2023-12-13T23:55:13.737621+0000","last_undegraded":"2023-12-13T23:55:13.737621+0000","last_fullsized":"2023-12-13T23:55:13.737621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613672+0000","last_change":"2023-12-13T23:55:11.761235+0000","last_active":"2023-12-13T23:55:14.613672+0000","last_peered":"2023-12-13T23:55:14.613672+0000","last_clean":"2023-12-13T23:55:14.613672+0000","last_became_active":"2023-12-13T23:55:11.760961+0000","last_became_peered":"2023-12-13T23:55:11.760961+0000","last_unstale":"2023-12-13T23:55:14.613672+0000","last_undegraded":"2023-12-13T23:55:14.613672+0000","last_fullsized":"2023-12-13T23:55:14.613672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737873+0000","last_change":"2023-12-13T23:55:12.757945+0000","last_active":"2023-12-13T23:55:13.737873+0000","last_peered":"2023-12-13T23:55:13.737873+0000","last_clean":"2023-12-13T23:55:13.737873+0000","last_became_active":"2023-12-13T23:55:12.757685+0000","last_became_peered":"2023-12-13T23:55:12.757685+0000","last_unstale":"2023-12-13T23:55:13.737873+0000","last_undegraded":"2023-12-13T23:55:13.737873+0000","last_fullsized":"2023-12-13T23:55:13.737873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740608+0000","last_change":"2023-12-13T23:55:11.769857+0000","last_active":"2023-12-13T23:55:13.740608+0000","last_peered":"2023-12-13T23:55:13.740608+0000","last_clean":"2023-12-13T23:55:13.740608+0000","last_became_active":"2023-12-13T23:55:11.769503+0000","last_became_peered":"2023-12-13T23:55:11.769503+0000","last_unstale":"2023-12-13T23:55:13.740608+0000","last_undegraded":"2023-12-13T23:55:13.740608+0000","last_fullsized":"2023-12-13T23:55:13.740608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624730+0000","last_change":"2023-12-13T23:55:12.747991+0000","last_active":"2023-12-13T23:55:14.624730+0000","last_peered":"2023-12-13T23:55:14.624730+0000","last_clean":"2023-12-13T23:55:14.624730+0000","last_became_active":"2023-12-13T23:55:12.746787+0000","last_became_peered":"2023-12-13T23:55:12.746787+0000","last_unstale":"2023-12-13T23:55:14.624730+0000","last_undegraded":"2023-12-13T23:55:14.624730+0000","last_fullsized":"2023-12-13T23:55:14.624730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624966+0000","last_change":"2023-12-13T23:55:12.755883+0000","last_active":"2023-12-13T23:55:14.624966+0000","last_peered":"2023-12-13T23:55:14.624966+0000","last_clean":"2023-12-13T23:55:14.624966+0000","last_became_active":"2023-12-13T23:55:12.755573+0000","last_became_peered":"2023-12-13T23:55:12.755573+0000","last_unstale":"2023-12-13T23:55:14.624966+0000","last_undegraded":"2023-12-13T23:55:14.624966+0000","last_fullsized":"2023-12-13T23:55:14.624966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741033+0000","last_change":"2023-12-13T23:55:11.769532+0000","last_active":"2023-12-13T23:55:13.741033+0000","last_peered":"2023-12-13T23:55:13.741033+0000","last_clean":"2023-12-13T23:55:13.741033+0000","last_became_active":"2023-12-13T23:55:11.769251+0000","last_became_peered":"2023-12-13T23:55:11.769251+0000","last_unstale":"2023-12-13T23:55:13.741033+0000","last_undegraded":"2023-12-13T23:55:13.741033+0000","last_fullsized":"2023-12-13T23:55:13.741033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624446+0000","last_change":"2023-12-13T23:55:12.757891+0000","last_active":"2023-12-13T23:55:14.624446+0000","last_peered":"2023-12-13T23:55:14.624446+0000","last_clean":"2023-12-13T23:55:14.624446+0000","last_became_active":"2023-12-13T23:55:12.757643+0000","last_became_peered":"2023-12-13T23:55:12.757643+0000","last_unstale":"2023-12-13T23:55:14.624446+0000","last_undegraded":"2023-12-13T23:55:14.624446+0000","last_fullsized":"2023-12-13T23:55:14.624446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740911+0000","last_change":"2023-12-13T23:55:11.769811+0000","last_active":"2023-12-13T23:55:13.740911+0000","last_peered":"2023-12-13T23:55:13.740911+0000","last_clean":"2023-12-13T23:55:13.740911+0000","last_became_active":"2023-12-13T23:55:11.769483+0000","last_became_peered":"2023-12-13T23:55:11.769483+0000","last_unstale":"2023-12-13T23:55:13.740911+0000","last_undegraded":"2023-12-13T23:55:13.740911+0000","last_fullsized":"2023-12-13T23:55:13.740911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624275+0000","last_change":"2023-12-13T23:55:12.758511+0000","last_active":"2023-12-13T23:55:14.624275+0000","last_peered":"2023-12-13T23:55:14.624275+0000","last_clean":"2023-12-13T23:55:14.624275+0000","last_became_active":"2023-12-13T23:55:12.757824+0000","last_became_peered":"2023-12-13T23:55:12.757824+0000","last_unstale":"2023-12-13T23:55:14.624275+0000","last_undegraded":"2023-12-13T23:55:14.624275+0000","last_fullsized":"2023-12-13T23:55:14.624275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737890+0000","last_change":"2023-12-13T23:55:11.761808+0000","last_active":"2023-12-13T23:55:13.737890+0000","last_peered":"2023-12-13T23:55:13.737890+0000","last_clean":"2023-12-13T23:55:13.737890+0000","last_became_active":"2023-12-13T23:55:11.761559+0000","last_became_peered":"2023-12-13T23:55:11.761559+0000","last_unstale":"2023-12-13T23:55:13.737890+0000","last_undegraded":"2023-12-13T23:55:13.737890+0000","last_fullsized":"2023-12-13T23:55:13.737890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737814+0000","last_change":"2023-12-13T23:55:12.748462+0000","last_active":"2023-12-13T23:55:13.737814+0000","last_peered":"2023-12-13T23:55:13.737814+0000","last_clean":"2023-12-13T23:55:13.737814+0000","last_became_active":"2023-12-13T23:55:12.748166+0000","last_became_peered":"2023-12-13T23:55:12.748166+0000","last_unstale":"2023-12-13T23:55:13.737814+0000","last_undegraded":"2023-12-13T23:55:13.737814+0000","last_fullsized":"2023-12-13T23:55:13.737814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624331+0000","last_change":"2023-12-13T23:55:11.771111+0000","last_active":"2023-12-13T23:55:14.624331+0000","last_peered":"2023-12-13T23:55:14.624331+0000","last_clean":"2023-12-13T23:55:14.624331+0000","last_became_active":"2023-12-13T23:55:11.770795+0000","last_became_peered":"2023-12-13T23:55:11.770795+0000","last_unstale":"2023-12-13T23:55:14.624331+0000","last_undegraded":"2023-12-13T23:55:14.624331+0000","last_fullsized":"2023-12-13T23:55:14.624331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:16.378 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-12-13T23:55:16.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.514+0000 7f1b347c9640 1 Processor -- start 2023-12-13T23:55:16.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.514+0000 7f1b347c9640 1 -- start start 2023-12-13T23:55:16.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b347c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1b2c0b0cf0 0x7f1b2c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b347c9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c161bb0 con 0x7f1b2c146aa0 2023-12-13T23:55:16.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b347c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b2c0b2850 con 0x7f1b2c0b3ec0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b347c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b2c0b16c0 con 0x7f1b2c0b0cf0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b32565640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f1b2c146aa0 0x7f1b2c0b2140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46726/0 (socket says 172.21.15.124:46726) 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b32565640 1 -- 172.21.15.124:0/197722796 learned_addr learned my addr 172.21.15.124:0/197722796 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912136982 0 0) 0x7f1b2c161bb0 con 0x7f1b2c146aa0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b31d64640 1 --2- 172.21.15.124:0/197722796 >> v2:172.21.15.203:3300/0 conn(0x7f1b2c0b0cf0 0x7f1b2c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b18003520 con 0x7f1b2c146aa0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.515+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945289884 0 0) 0x7f1b18003520 con 0x7f1b2c146aa0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 >> v2:172.21.15.203:3300/0 conn(0x7f1b2c0b0cf0 msgr2=0x7f1b2c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 --2- 172.21.15.124:0/197722796 >> v2:172.21.15.203:3300/0 conn(0x7f1b2c0b0cf0 0x7f1b2c0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 >> v1:172.21.15.203:6789/0 conn(0x7f1b2c0b3ec0 legacy=0x7f1b2c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c160fe0 con 0x7f1b2c146aa0 2023-12-13T23:55:16.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f1b280019f0 con 0x7f1b2c146aa0 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 >> v1:172.21.15.124:6789/0 conn(0x7f1b2c146aa0 legacy=0x7f1b2c0b2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 0x7f1b18003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b180047c0 0x7f1b18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b180052a0 0x7f1b1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c0b16c0 con 0x7f1b18003970 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1800c150 con 0x7f1b180052a0 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1800c310 con 0x7f1b180047c0 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 0x7f1b18003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31d64640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b180047c0 0x7f1b18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32d66640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b180052a0 0x7f1b1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 -- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b180047c0 msgr2=0x7f1b18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b180047c0 0x7f1b18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 -- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b180052a0 msgr2=0x7f1b1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b180052a0 0x7f1b1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32565640 1 -- 172.21.15.124:0/197722796 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18003520 con 0x7f1b18003970 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b32d66640 1 -- 172.21.15.124:0/197722796 reap_dead start 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.516+0000 7f1b31d64640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b180047c0 0x7f1b18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:16.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b32565640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 0x7f1b18003d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800a810 tx=0x7f1b180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a98e2572c05f9cb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28034630 con 0x7f1b18003970 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b2802f070 con 0x7f1b18003970 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b31563640 1 -- 172.21.15.124:0/197722796 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b28034a10 con 0x7f1b18003970 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 -- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 msgr2=0x7f1b18003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 0x7f1b18003d70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1b2800a810 tx=0x7f1b180057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 -- 172.21.15.124:0/197722796 shutdown_connections 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 --2- 172.21.15.124:0/197722796 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b18003970 0x7f1b18003d70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 -- 172.21.15.124:0/197722796 >> 172.21.15.124:0/197722796 conn(0x7f1b2c06a4d0 msgr2=0x7f1b2c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 -- 172.21.15.124:0/197722796 shutdown_connections 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.517+0000 7f1b347c9640 1 -- 172.21.15.124:0/197722796 wait complete. 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.518+0000 7f1b347c9640 1 Processor -- start 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 -- start start 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 0x7f1b2c141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b2c160070 0x7f1b2c15ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c15ce40 con 0x7f1b2c1604f0 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c159070 con 0x7f1b18003970 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b347c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2c15c230 con 0x7f1b2c160070 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b32d66640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b32d66640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48728/0 (socket says 172.21.15.124:48728) 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b32d66640 1 -- 172.21.15.124:0/951874051 learned_addr learned my addr 172.21.15.124:0/951874051 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b31d64640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b2c160070 0x7f1b2c15ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.519+0000 7f1b32565640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 0x7f1b2c141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 -- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b2c160070 msgr2=0x7f1b2c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b2c160070 0x7f1b2c15ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 -- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 msgr2=0x7f1b2c141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 0x7f1b2c141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 -- 172.21.15.124:0/951874051 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b2c15c510 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32565640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 0x7f1b2c141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b32d66640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0ad9a0 tx=0x7f1b2400e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b240054b0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b240069e0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2c15c7f0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b24006d90 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.520+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2c141d70 con 0x7f1b2c1604f0 2023-12-13T23:55:16.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.521+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b2c1593c0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.521+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1b24007030 con 0x7f1b2c1604f0 2023-12-13T23:55:16.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.522+0000 7f1b22ffd640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 0x7f1b1006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.522+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b24088ac0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.522+0000 7f1b32565640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 0x7f1b1006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.522+0000 7f1b32565640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 0x7f1b1006d6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b28034000 tx=0x7f1b2802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.526+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1b240538a0 con 0x7f1b2c1604f0 2023-12-13T23:55:16.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.755+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f1b2c069400 con 0x7f1b2c1604f0 2023-12-13T23:55:16.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.756+0000 7f1b22ffd640 1 -- 172.21.15.124:0/951874051 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1021 (secure 0 0 0) 0x7f1b24019410 con 0x7f1b2c1604f0 2023-12-13T23:55:16.754 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:16.754 INFO:teuthology.orchestra.run.smithi124.stdout:{"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":28,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":21,"num_osds":8,"num_up_osds":8,"osd_up_since":1702511699,"num_in_osds":8,"osd_in_since":1702511690,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":96},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":224653312,"bytes_avail":772869459968,"bytes_total":773094113280,"unknown_pgs_ratio":0.98969072103500366},"fsmap":{"epoch":6,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4308}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-12-13T23:54:47.989094+0000","services":{}},"progress_events":{}} 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 msgr2=0x7f1b1006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 0x7f1b1006d6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b28034000 tx=0x7f1b2802f000 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 msgr2=0x7f1b2c15e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0ad9a0 tx=0x7f1b2400e8a0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 shutdown_connections 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1b1006b1f0 0x7f1b1006d6b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1b2c1604f0 0x7f1b2c15e2f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1b2c160070 0x7f1b2c15ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 --2- 172.21.15.124:0/951874051 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1b18003970 0x7f1b2c141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 >> 172.21.15.124:0/951874051 conn(0x7f1b2c06a4d0 msgr2=0x7f1b2c0b0210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 shutdown_connections 2023-12-13T23:55:16.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.757+0000 7f1b347c9640 1 -- 172.21.15.124:0/951874051 wait complete. 2023-12-13T23:55:16.766 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-12-13T23:55:16.766 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:16.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.867+0000 7f6821231640 1 Processor -- start 2023-12-13T23:55:16.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.867+0000 7f6821231640 1 -- start start 2023-12-13T23:55:16.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f6821231640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f681c15b130 0x7f681c0d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f6821231640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f681c15f760 0x7f681c0cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f6821231640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0cf560 con 0x7f681c0cf150 2023-12-13T23:55:16.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f6821231640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c15fd20 con 0x7f681c15b130 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f6821231640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c0d0730 con 0x7f681c15f760 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f681ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f681c15b130 0x7f681c0d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f681b59e640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f681c0cf150 0x7f681c06ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:46732/0 (socket says 172.21.15.124:46732) 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f681ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f681c15b130 0x7f681c0d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48738/0 (socket says 172.21.15.124:48738) 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.868+0000 7f681b59e640 1 -- 172.21.15.124:0/3062569633 learned_addr learned my addr 172.21.15.124:0/3062569633 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f681ad9d640 1 -- 172.21.15.124:0/3062569633 >> v2:172.21.15.203:3300/0 conn(0x7f681c15f760 msgr2=0x7f681c0cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f681ad9d640 1 --2- 172.21.15.124:0/3062569633 >> v2:172.21.15.203:3300/0 conn(0x7f681c15f760 0x7f681c0cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f681ad9d640 1 -- 172.21.15.124:0/3062569633 >> v1:172.21.15.124:6789/0 conn(0x7f681c0cf150 legacy=0x7f681c06ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f681ad9d640 1 -- 172.21.15.124:0/3062569633 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c0b3f10 con 0x7f681c15b130 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f6819d9b640 1 -- 172.21.15.124:0/3062569633 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650121013 0 0) 0x7f681c0cf560 con 0x7f681c0cf150 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f681ad9d640 1 --2- 172.21.15.124:0/3062569633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c15b130 0x7f681c0d01f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f681c0b71b0 tx=0x7f680c02be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c4ea73f8e6d331 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f6819d9b640 1 -- 172.21.15.124:0/3062569633 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c02cbd0 con 0x7f681c15b130 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f6819d9b640 1 -- 172.21.15.124:0/3062569633 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c02cd30 con 0x7f681c15b130 2023-12-13T23:55:16.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.869+0000 7f6819d9b640 1 -- 172.21.15.124:0/3062569633 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c02cea0 con 0x7f681c15b130 2023-12-13T23:55:16.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 -- 172.21.15.124:0/3062569633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c15b130 msgr2=0x7f681c0d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 --2- 172.21.15.124:0/3062569633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c15b130 0x7f681c0d01f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f681c0b71b0 tx=0x7f680c02be20 comp rx=0 tx=0).stop 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 -- 172.21.15.124:0/3062569633 shutdown_connections 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 --2- 172.21.15.124:0/3062569633 >> v2:172.21.15.203:3300/0 conn(0x7f681c15f760 0x7f681c0cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 --2- 172.21.15.124:0/3062569633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c15b130 0x7f681c0d01f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 -- 172.21.15.124:0/3062569633 >> 172.21.15.124:0/3062569633 conn(0x7f681c0b2c10 msgr2=0x7f681c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 -- 172.21.15.124:0/3062569633 shutdown_connections 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.870+0000 7f6821231640 1 -- 172.21.15.124:0/3062569633 wait complete. 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.871+0000 7f6821231640 1 Processor -- start 2023-12-13T23:55:16.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.871+0000 7f6821231640 1 -- start start 2023-12-13T23:55:16.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f681c15b130 0x7f681c1f0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c15f760 0x7f681c1f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c8ab0 con 0x7f681c0cf150 2023-12-13T23:55:16.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c8d90 con 0x7f681c15f760 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f6821231640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0c9070 con 0x7f681c15b130 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48754/0 (socket says 172.21.15.124:48754) 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 -- 172.21.15.124:0/1641326912 learned_addr learned my addr 172.21.15.124:0/1641326912 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 -- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f681c15b130 msgr2=0x7f681c1f0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f681c15b130 0x7f681c1f0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 -- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c15f760 msgr2=0x7f681c1f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c15f760 0x7f681c1f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.872+0000 7f681ad9d640 1 -- 172.21.15.124:0/1641326912 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c009a70 con 0x7f681c0cf150 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f681ad9d640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f680c009d90 tx=0x7f680c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c02cea0 con 0x7f681c0cf150 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c03e610 con 0x7f681c0cf150 2023-12-13T23:55:16.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c03e9c0 con 0x7f681c0cf150 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c0c9350 con 0x7f681c0cf150 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.873+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c156950 con 0x7f681c0cf150 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.874+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c067fc0 con 0x7f681c0cf150 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.874+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f680c03b050 con 0x7f681c0cf150 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.875+0000 7f68037fe640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 0x7f680806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:16.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.875+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f680c0ac3a0 con 0x7f681c0cf150 2023-12-13T23:55:16.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.875+0000 7f681a59c640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 0x7f680806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:16.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.876+0000 7f681a59c640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 0x7f680806d680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f681c04ed80 tx=0x7f6810006bf0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:16.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:16.879+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f680c0770d0 con 0x7f681c0cf150 2023-12-13T23:55:17.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.043+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f681c0cb690 con 0x7f680806b1c0 2023-12-13T23:55:17.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.050+0000 7f68037fe640 1 -- 172.21.15.124:0/1641326912 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264701 (secure 0 0 0) 0x7f6810012390 con 0x7f680806b1c0 2023-12-13T23:55:17.051 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:17.052 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 msgr2=0x7f680806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 0x7f680806d680 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f681c04ed80 tx=0x7f6810006bf0 comp rx=0 tx=0).stop 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 msgr2=0x7f681c0c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f680c009d90 tx=0x7f680c004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 shutdown_connections 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f680806b1c0 0x7f680806d680 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f681c15f760 0x7f681c1f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:17.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f681c15b130 0x7f681c1f0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:17.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 --2- 172.21.15.124:0/1641326912 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f681c0cf150 0x7f681c0c8110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:17.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 >> 172.21.15.124:0/1641326912 conn(0x7f681c0b2c10 msgr2=0x7f681c15ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:17.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 shutdown_connections 2023-12-13T23:55:17.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:17.053+0000 7f6821231640 1 -- 172.21.15.124:0/1641326912 wait complete. 2023-12-13T23:55:17.062 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2023-12-13T23:55:16.109984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2928640,"data_stored":1758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.367307"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741160+0000","last_change":"2023-12-13T23:55:12.749065+0000","last_active":"2023-12-13T23:55:13.741160+0000","last_peered":"2023-12-13T23:55:13.741160+0000","last_clean":"2023-12-13T23:55:13.741160+0000","last_became_active":"2023-12-13T23:55:12.748399+0000","last_became_peered":"2023-12-13T23:55:12.748399+0000","last_unstale":"2023-12-13T23:55:13.741160+0000","last_undegraded":"2023-12-13T23:55:13.741160+0000","last_fullsized":"2023-12-13T23:55:13.741160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613725+0000","last_change":"2023-12-13T23:55:12.751938+0000","last_active":"2023-12-13T23:55:14.613725+0000","last_peered":"2023-12-13T23:55:14.613725+0000","last_clean":"2023-12-13T23:55:14.613725+0000","last_became_active":"2023-12-13T23:55:12.751688+0000","last_became_peered":"2023-12-13T23:55:12.751688+0000","last_unstale":"2023-12-13T23:55:14.613725+0000","last_undegraded":"2023-12-13T23:55:14.613725+0000","last_fullsized":"2023-12-13T23:55:14.613725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624101+0000","last_change":"2023-12-13T23:55:12.757831+0000","last_active":"2023-12-13T23:55:14.624101+0000","last_peered":"2023-12-13T23:55:14.624101+0000","last_clean":"2023-12-13T23:55:14.624101+0000","last_became_active":"2023-12-13T23:55:12.757403+0000","last_became_peered":"2023-12-13T23:55:12.757403+0000","last_unstale":"2023-12-13T23:55:14.624101+0000","last_undegraded":"2023-12-13T23:55:14.624101+0000","last_fullsized":"2023-12-13T23:55:14.624101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624401+0000","last_change":"2023-12-13T23:55:12.757838+0000","last_active":"2023-12-13T23:55:14.624401+0000","last_peered":"2023-12-13T23:55:14.624401+0000","last_clean":"2023-12-13T23:55:14.624401+0000","last_became_active":"2023-12-13T23:55:12.757590+0000","last_became_peered":"2023-12-13T23:55:12.757590+0000","last_unstale":"2023-12-13T23:55:14.624401+0000","last_undegraded":"2023-12-13T23:55:14.624401+0000","last_fullsized":"2023-12-13T23:55:14.624401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740506+0000","last_change":"2023-12-13T23:55:12.750174+0000","last_active":"2023-12-13T23:55:13.740506+0000","last_peered":"2023-12-13T23:55:13.740506+0000","last_clean":"2023-12-13T23:55:13.740506+0000","last_became_active":"2023-12-13T23:55:12.749937+0000","last_became_peered":"2023-12-13T23:55:12.749937+0000","last_unstale":"2023-12-13T23:55:13.740506+0000","last_undegraded":"2023-12-13T23:55:13.740506+0000","last_fullsized":"2023-12-13T23:55:13.740506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624775+0000","last_change":"2023-12-13T23:55:12.760175+0000","last_active":"2023-12-13T23:55:14.624775+0000","last_peered":"2023-12-13T23:55:14.624775+0000","last_clean":"2023-12-13T23:55:14.624775+0000","last_became_active":"2023-12-13T23:55:12.759888+0000","last_became_peered":"2023-12-13T23:55:12.759888+0000","last_unstale":"2023-12-13T23:55:14.624775+0000","last_undegraded":"2023-12-13T23:55:14.624775+0000","last_fullsized":"2023-12-13T23:55:14.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740922+0000","last_change":"2023-12-13T23:55:12.758545+0000","last_active":"2023-12-13T23:55:13.740922+0000","last_peered":"2023-12-13T23:55:13.740922+0000","last_clean":"2023-12-13T23:55:13.740922+0000","last_became_active":"2023-12-13T23:55:12.758292+0000","last_became_peered":"2023-12-13T23:55:12.758292+0000","last_unstale":"2023-12-13T23:55:13.740922+0000","last_undegraded":"2023-12-13T23:55:13.740922+0000","last_fullsized":"2023-12-13T23:55:13.740922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737487+0000","last_change":"2023-12-13T23:55:12.756050+0000","last_active":"2023-12-13T23:55:13.737487+0000","last_peered":"2023-12-13T23:55:13.737487+0000","last_clean":"2023-12-13T23:55:13.737487+0000","last_became_active":"2023-12-13T23:55:12.755765+0000","last_became_peered":"2023-12-13T23:55:12.755765+0000","last_unstale":"2023-12-13T23:55:13.737487+0000","last_undegraded":"2023-12-13T23:55:13.737487+0000","last_fullsized":"2023-12-13T23:55:13.737487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741293+0000","last_change":"2023-12-13T23:55:12.751022+0000","last_active":"2023-12-13T23:55:13.741293+0000","last_peered":"2023-12-13T23:55:13.741293+0000","last_clean":"2023-12-13T23:55:13.741293+0000","last_became_active":"2023-12-13T23:55:12.750750+0000","last_became_peered":"2023-12-13T23:55:12.750750+0000","last_unstale":"2023-12-13T23:55:13.741293+0000","last_undegraded":"2023-12-13T23:55:13.741293+0000","last_fullsized":"2023-12-13T23:55:13.741293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624534+0000","last_change":"2023-12-13T23:55:12.754201+0000","last_active":"2023-12-13T23:55:14.624534+0000","last_peered":"2023-12-13T23:55:14.624534+0000","last_clean":"2023-12-13T23:55:14.624534+0000","last_became_active":"2023-12-13T23:55:12.753905+0000","last_became_peered":"2023-12-13T23:55:12.753905+0000","last_unstale":"2023-12-13T23:55:14.624534+0000","last_undegraded":"2023-12-13T23:55:14.624534+0000","last_fullsized":"2023-12-13T23:55:14.624534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624445+0000","last_change":"2023-12-13T23:55:12.756766+0000","last_active":"2023-12-13T23:55:14.624445+0000","last_peered":"2023-12-13T23:55:14.624445+0000","last_clean":"2023-12-13T23:55:14.624445+0000","last_became_active":"2023-12-13T23:55:12.756500+0000","last_became_peered":"2023-12-13T23:55:12.756500+0000","last_unstale":"2023-12-13T23:55:14.624445+0000","last_undegraded":"2023-12-13T23:55:14.624445+0000","last_fullsized":"2023-12-13T23:55:14.624445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741255+0000","last_change":"2023-12-13T23:55:12.756454+0000","last_active":"2023-12-13T23:55:13.741255+0000","last_peered":"2023-12-13T23:55:13.741255+0000","last_clean":"2023-12-13T23:55:13.741255+0000","last_became_active":"2023-12-13T23:55:12.756151+0000","last_became_peered":"2023-12-13T23:55:12.756151+0000","last_unstale":"2023-12-13T23:55:13.741255+0000","last_undegraded":"2023-12-13T23:55:13.741255+0000","last_fullsized":"2023-12-13T23:55:13.741255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:41.266746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737767+0000","last_change":"2023-12-13T23:55:12.751798+0000","last_active":"2023-12-13T23:55:13.737767+0000","last_peered":"2023-12-13T23:55:13.737767+0000","last_clean":"2023-12-13T23:55:13.737767+0000","last_became_active":"2023-12-13T23:55:12.751513+0000","last_became_peered":"2023-12-13T23:55:12.751513+0000","last_unstale":"2023-12-13T23:55:13.737767+0000","last_undegraded":"2023-12-13T23:55:13.737767+0000","last_fullsized":"2023-12-13T23:55:13.737767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624265+0000","last_change":"2023-12-13T23:55:12.757612+0000","last_active":"2023-12-13T23:55:14.624265+0000","last_peered":"2023-12-13T23:55:14.624265+0000","last_clean":"2023-12-13T23:55:14.624265+0000","last_became_active":"2023-12-13T23:55:12.757320+0000","last_became_peered":"2023-12-13T23:55:12.757320+0000","last_unstale":"2023-12-13T23:55:14.624265+0000","last_undegraded":"2023-12-13T23:55:14.624265+0000","last_fullsized":"2023-12-13T23:55:14.624265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613523+0000","last_change":"2023-12-13T23:55:12.743417+0000","last_active":"2023-12-13T23:55:14.613523+0000","last_peered":"2023-12-13T23:55:14.613523+0000","last_clean":"2023-12-13T23:55:14.613523+0000","last_became_active":"2023-12-13T23:55:12.743121+0000","last_became_peered":"2023-12-13T23:55:12.743121+0000","last_unstale":"2023-12-13T23:55:14.613523+0000","last_undegraded":"2023-12-13T23:55:14.613523+0000","last_fullsized":"2023-12-13T23:55:14.613523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624106+0000","last_change":"2023-12-13T23:55:12.761124+0000","last_active":"2023-12-13T23:55:14.624106+0000","last_peered":"2023-12-13T23:55:14.624106+0000","last_clean":"2023-12-13T23:55:14.624106+0000","last_became_active":"2023-12-13T23:55:12.760661+0000","last_became_peered":"2023-12-13T23:55:12.760661+0000","last_unstale":"2023-12-13T23:55:14.624106+0000","last_undegraded":"2023-12-13T23:55:14.624106+0000","last_fullsized":"2023-12-13T23:55:14.624106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624905+0000","last_change":"2023-12-13T23:55:12.752470+0000","last_active":"2023-12-13T23:55:14.624905+0000","last_peered":"2023-12-13T23:55:14.624905+0000","last_clean":"2023-12-13T23:55:14.624905+0000","last_became_active":"2023-12-13T23:55:12.752197+0000","last_became_peered":"2023-12-13T23:55:12.752197+0000","last_unstale":"2023-12-13T23:55:14.624905+0000","last_undegraded":"2023-12-13T23:55:14.624905+0000","last_fullsized":"2023-12-13T23:55:14.624905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623990+0000","last_change":"2023-12-13T23:55:12.748844+0000","last_active":"2023-12-13T23:55:14.623990+0000","last_peered":"2023-12-13T23:55:14.623990+0000","last_clean":"2023-12-13T23:55:14.623990+0000","last_became_active":"2023-12-13T23:55:12.748564+0000","last_became_peered":"2023-12-13T23:55:12.748564+0000","last_unstale":"2023-12-13T23:55:14.623990+0000","last_undegraded":"2023-12-13T23:55:14.623990+0000","last_fullsized":"2023-12-13T23:55:14.623990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741142+0000","last_change":"2023-12-13T23:55:12.749584+0000","last_active":"2023-12-13T23:55:13.741142+0000","last_peered":"2023-12-13T23:55:13.741142+0000","last_clean":"2023-12-13T23:55:13.741142+0000","last_became_active":"2023-12-13T23:55:12.749330+0000","last_became_peered":"2023-12-13T23:55:12.749330+0000","last_unstale":"2023-12-13T23:55:13.741142+0000","last_undegraded":"2023-12-13T23:55:13.741142+0000","last_fullsized":"2023-12-13T23:55:13.741142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624156+0000","last_change":"2023-12-13T23:55:12.746390+0000","last_active":"2023-12-13T23:55:14.624156+0000","last_peered":"2023-12-13T23:55:14.624156+0000","last_clean":"2023-12-13T23:55:14.624156+0000","last_became_active":"2023-12-13T23:55:12.746135+0000","last_became_peered":"2023-12-13T23:55:12.746135+0000","last_unstale":"2023-12-13T23:55:14.624156+0000","last_undegraded":"2023-12-13T23:55:14.624156+0000","last_fullsized":"2023-12-13T23:55:14.624156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624034+0000","last_change":"2023-12-13T23:55:12.752805+0000","last_active":"2023-12-13T23:55:14.624034+0000","last_peered":"2023-12-13T23:55:14.624034+0000","last_clean":"2023-12-13T23:55:14.624034+0000","last_became_active":"2023-12-13T23:55:12.752447+0000","last_became_peered":"2023-12-13T23:55:12.752447+0000","last_unstale":"2023-12-13T23:55:14.624034+0000","last_undegraded":"2023-12-13T23:55:14.624034+0000","last_fullsized":"2023-12-13T23:55:14.624034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:24.291337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624917+0000","last_change":"2023-12-13T23:55:12.754499+0000","last_active":"2023-12-13T23:55:14.624917+0000","last_peered":"2023-12-13T23:55:14.624917+0000","last_clean":"2023-12-13T23:55:14.624917+0000","last_became_active":"2023-12-13T23:55:12.753996+0000","last_became_peered":"2023-12-13T23:55:12.753996+0000","last_unstale":"2023-12-13T23:55:14.624917+0000","last_undegraded":"2023-12-13T23:55:14.624917+0000","last_fullsized":"2023-12-13T23:55:14.624917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737534+0000","last_change":"2023-12-13T23:55:12.750598+0000","last_active":"2023-12-13T23:55:13.737534+0000","last_peered":"2023-12-13T23:55:13.737534+0000","last_clean":"2023-12-13T23:55:13.737534+0000","last_became_active":"2023-12-13T23:55:12.750296+0000","last_became_peered":"2023-12-13T23:55:12.750296+0000","last_unstale":"2023-12-13T23:55:13.737534+0000","last_undegraded":"2023-12-13T23:55:13.737534+0000","last_fullsized":"2023-12-13T23:55:13.737534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624327+0000","last_change":"2023-12-13T23:55:12.750435+0000","last_active":"2023-12-13T23:55:14.624327+0000","last_peered":"2023-12-13T23:55:14.624327+0000","last_clean":"2023-12-13T23:55:14.624327+0000","last_became_active":"2023-12-13T23:55:12.750151+0000","last_became_peered":"2023-12-13T23:55:12.750151+0000","last_unstale":"2023-12-13T23:55:14.624327+0000","last_undegraded":"2023-12-13T23:55:14.624327+0000","last_fullsized":"2023-12-13T23:55:14.624327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737900+0000","last_change":"2023-12-13T23:55:12.750495+0000","last_active":"2023-12-13T23:55:13.737900+0000","last_peered":"2023-12-13T23:55:13.737900+0000","last_clean":"2023-12-13T23:55:13.737900+0000","last_became_active":"2023-12-13T23:55:12.750227+0000","last_became_peered":"2023-12-13T23:55:12.750227+0000","last_unstale":"2023-12-13T23:55:13.737900+0000","last_undegraded":"2023-12-13T23:55:13.737900+0000","last_fullsized":"2023-12-13T23:55:13.737900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624667+0000","last_change":"2023-12-13T23:55:12.751349+0000","last_active":"2023-12-13T23:55:14.624667+0000","last_peered":"2023-12-13T23:55:14.624667+0000","last_clean":"2023-12-13T23:55:14.624667+0000","last_became_active":"2023-12-13T23:55:12.751057+0000","last_became_peered":"2023-12-13T23:55:12.751057+0000","last_unstale":"2023-12-13T23:55:14.624667+0000","last_undegraded":"2023-12-13T23:55:14.624667+0000","last_fullsized":"2023-12-13T23:55:14.624667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613647+0000","last_change":"2023-12-13T23:55:12.748079+0000","last_active":"2023-12-13T23:55:14.613647+0000","last_peered":"2023-12-13T23:55:14.613647+0000","last_clean":"2023-12-13T23:55:14.613647+0000","last_became_active":"2023-12-13T23:55:12.747722+0000","last_became_peered":"2023-12-13T23:55:12.747722+0000","last_unstale":"2023-12-13T23:55:14.613647+0000","last_undegraded":"2023-12-13T23:55:14.613647+0000","last_fullsized":"2023-12-13T23:55:14.613647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737802+0000","last_change":"2023-12-13T23:55:12.747671+0000","last_active":"2023-12-13T23:55:13.737802+0000","last_peered":"2023-12-13T23:55:13.737802+0000","last_clean":"2023-12-13T23:55:13.737802+0000","last_became_active":"2023-12-13T23:55:12.747431+0000","last_became_peered":"2023-12-13T23:55:12.747431+0000","last_unstale":"2023-12-13T23:55:13.737802+0000","last_undegraded":"2023-12-13T23:55:13.737802+0000","last_fullsized":"2023-12-13T23:55:13.737802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624186+0000","last_change":"2023-12-13T23:55:12.751994+0000","last_active":"2023-12-13T23:55:14.624186+0000","last_peered":"2023-12-13T23:55:14.624186+0000","last_clean":"2023-12-13T23:55:14.624186+0000","last_became_active":"2023-12-13T23:55:12.751598+0000","last_became_peered":"2023-12-13T23:55:12.751598+0000","last_unstale":"2023-12-13T23:55:14.624186+0000","last_undegraded":"2023-12-13T23:55:14.624186+0000","last_fullsized":"2023-12-13T23:55:14.624186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624938+0000","last_change":"2023-12-13T23:55:12.754312+0000","last_active":"2023-12-13T23:55:14.624938+0000","last_peered":"2023-12-13T23:55:14.624938+0000","last_clean":"2023-12-13T23:55:14.624938+0000","last_became_active":"2023-12-13T23:55:12.753618+0000","last_became_peered":"2023-12-13T23:55:12.753618+0000","last_unstale":"2023-12-13T23:55:14.624938+0000","last_undegraded":"2023-12-13T23:55:14.624938+0000","last_fullsized":"2023-12-13T23:55:14.624938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623822+0000","last_change":"2023-12-13T23:55:12.755004+0000","last_active":"2023-12-13T23:55:14.623822+0000","last_peered":"2023-12-13T23:55:14.623822+0000","last_clean":"2023-12-13T23:55:14.623822+0000","last_became_active":"2023-12-13T23:55:12.754761+0000","last_became_peered":"2023-12-13T23:55:12.754761+0000","last_unstale":"2023-12-13T23:55:14.623822+0000","last_undegraded":"2023-12-13T23:55:14.623822+0000","last_fullsized":"2023-12-13T23:55:14.623822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624213+0000","last_change":"2023-12-13T23:55:12.758828+0000","last_active":"2023-12-13T23:55:14.624213+0000","last_peered":"2023-12-13T23:55:14.624213+0000","last_clean":"2023-12-13T23:55:14.624213+0000","last_became_active":"2023-12-13T23:55:12.758520+0000","last_became_peered":"2023-12-13T23:55:12.758520+0000","last_unstale":"2023-12-13T23:55:14.624213+0000","last_undegraded":"2023-12-13T23:55:14.624213+0000","last_fullsized":"2023-12-13T23:55:14.624213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740370+0000","last_change":"2023-12-13T23:55:12.745882+0000","last_active":"2023-12-13T23:55:13.740370+0000","last_peered":"2023-12-13T23:55:13.740370+0000","last_clean":"2023-12-13T23:55:13.740370+0000","last_became_active":"2023-12-13T23:55:12.744988+0000","last_became_peered":"2023-12-13T23:55:12.744988+0000","last_unstale":"2023-12-13T23:55:13.740370+0000","last_undegraded":"2023-12-13T23:55:13.740370+0000","last_fullsized":"2023-12-13T23:55:13.740370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:57.837875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737851+0000","last_change":"2023-12-13T23:55:11.769713+0000","last_active":"2023-12-13T23:55:13.737851+0000","last_peered":"2023-12-13T23:55:13.737851+0000","last_clean":"2023-12-13T23:55:13.737851+0000","last_became_active":"2023-12-13T23:55:11.769540+0000","last_became_peered":"2023-12-13T23:55:11.769540+0000","last_unstale":"2023-12-13T23:55:13.737851+0000","last_undegraded":"2023-12-13T23:55:13.737851+0000","last_fullsized":"2023-12-13T23:55:13.737851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624177+0000","last_change":"2023-12-13T23:55:12.752916+0000","last_active":"2023-12-13T23:55:14.624177+0000","last_peered":"2023-12-13T23:55:14.624177+0000","last_clean":"2023-12-13T23:55:14.624177+0000","last_became_active":"2023-12-13T23:55:12.751877+0000","last_became_peered":"2023-12-13T23:55:12.751877+0000","last_unstale":"2023-12-13T23:55:14.624177+0000","last_undegraded":"2023-12-13T23:55:14.624177+0000","last_fullsized":"2023-12-13T23:55:14.624177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624064+0000","last_change":"2023-12-13T23:55:11.767168+0000","last_active":"2023-12-13T23:55:14.624064+0000","last_peered":"2023-12-13T23:55:14.624064+0000","last_clean":"2023-12-13T23:55:14.624064+0000","last_became_active":"2023-12-13T23:55:11.766944+0000","last_became_peered":"2023-12-13T23:55:11.766944+0000","last_unstale":"2023-12-13T23:55:14.624064+0000","last_undegraded":"2023-12-13T23:55:14.624064+0000","last_fullsized":"2023-12-13T23:55:14.624064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737587+0000","last_change":"2023-12-13T23:55:11.764302+0000","last_active":"2023-12-13T23:55:13.737587+0000","last_peered":"2023-12-13T23:55:13.737587+0000","last_clean":"2023-12-13T23:55:13.737587+0000","last_became_active":"2023-12-13T23:55:11.763930+0000","last_became_peered":"2023-12-13T23:55:11.763930+0000","last_unstale":"2023-12-13T23:55:13.737587+0000","last_undegraded":"2023-12-13T23:55:13.737587+0000","last_fullsized":"2023-12-13T23:55:13.737587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737560+0000","last_change":"2023-12-13T23:55:12.760219+0000","last_active":"2023-12-13T23:55:13.737560+0000","last_peered":"2023-12-13T23:55:13.737560+0000","last_clean":"2023-12-13T23:55:13.737560+0000","last_became_active":"2023-12-13T23:55:12.759890+0000","last_became_peered":"2023-12-13T23:55:12.759890+0000","last_unstale":"2023-12-13T23:55:13.737560+0000","last_undegraded":"2023-12-13T23:55:13.737560+0000","last_fullsized":"2023-12-13T23:55:13.737560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624404+0000","last_change":"2023-12-13T23:55:11.763128+0000","last_active":"2023-12-13T23:55:14.624404+0000","last_peered":"2023-12-13T23:55:14.624404+0000","last_clean":"2023-12-13T23:55:14.624404+0000","last_became_active":"2023-12-13T23:55:11.762848+0000","last_became_peered":"2023-12-13T23:55:11.762848+0000","last_unstale":"2023-12-13T23:55:14.624404+0000","last_undegraded":"2023-12-13T23:55:14.624404+0000","last_fullsized":"2023-12-13T23:55:14.624404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613510+0000","last_change":"2023-12-13T23:55:12.747367+0000","last_active":"2023-12-13T23:55:14.613510+0000","last_peered":"2023-12-13T23:55:14.613510+0000","last_clean":"2023-12-13T23:55:14.613510+0000","last_became_active":"2023-12-13T23:55:12.747074+0000","last_became_peered":"2023-12-13T23:55:12.747074+0000","last_unstale":"2023-12-13T23:55:14.613510+0000","last_undegraded":"2023-12-13T23:55:14.613510+0000","last_fullsized":"2023-12-13T23:55:14.613510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624348+0000","last_change":"2023-12-13T23:55:11.766140+0000","last_active":"2023-12-13T23:55:14.624348+0000","last_peered":"2023-12-13T23:55:14.624348+0000","last_clean":"2023-12-13T23:55:14.624348+0000","last_became_active":"2023-12-13T23:55:11.765848+0000","last_became_peered":"2023-12-13T23:55:11.765848+0000","last_unstale":"2023-12-13T23:55:14.624348+0000","last_undegraded":"2023-12-13T23:55:14.624348+0000","last_fullsized":"2023-12-13T23:55:14.624348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624786+0000","last_change":"2023-12-13T23:55:12.750678+0000","last_active":"2023-12-13T23:55:14.624786+0000","last_peered":"2023-12-13T23:55:14.624786+0000","last_clean":"2023-12-13T23:55:14.624786+0000","last_became_active":"2023-12-13T23:55:12.750200+0000","last_became_peered":"2023-12-13T23:55:12.750200+0000","last_unstale":"2023-12-13T23:55:14.624786+0000","last_undegraded":"2023-12-13T23:55:14.624786+0000","last_fullsized":"2023-12-13T23:55:14.624786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624863+0000","last_change":"2023-12-13T23:55:11.763844+0000","last_active":"2023-12-13T23:55:14.624863+0000","last_peered":"2023-12-13T23:55:14.624863+0000","last_clean":"2023-12-13T23:55:14.624863+0000","last_became_active":"2023-12-13T23:55:11.763574+0000","last_became_peered":"2023-12-13T23:55:11.763574+0000","last_unstale":"2023-12-13T23:55:14.624863+0000","last_undegraded":"2023-12-13T23:55:14.624863+0000","last_fullsized":"2023-12-13T23:55:14.624863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:41:34.696890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624340+0000","last_change":"2023-12-13T23:55:12.754675+0000","last_active":"2023-12-13T23:55:14.624340+0000","last_peered":"2023-12-13T23:55:14.624340+0000","last_clean":"2023-12-13T23:55:14.624340+0000","last_became_active":"2023-12-13T23:55:12.753740+0000","last_became_peered":"2023-12-13T23:55:12.753740+0000","last_unstale":"2023-12-13T23:55:14.624340+0000","last_undegraded":"2023-12-13T23:55:14.624340+0000","last_fullsized":"2023-12-13T23:55:14.624340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613654+0000","last_change":"2023-12-13T23:55:11.765823+0000","last_active":"2023-12-13T23:55:14.613654+0000","last_peered":"2023-12-13T23:55:14.613654+0000","last_clean":"2023-12-13T23:55:14.613654+0000","last_became_active":"2023-12-13T23:55:11.765515+0000","last_became_peered":"2023-12-13T23:55:11.765515+0000","last_unstale":"2023-12-13T23:55:14.613654+0000","last_undegraded":"2023-12-13T23:55:14.613654+0000","last_fullsized":"2023-12-13T23:55:14.613654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741390+0000","last_change":"2023-12-13T23:55:12.750387+0000","last_active":"2023-12-13T23:55:13.741390+0000","last_peered":"2023-12-13T23:55:13.741390+0000","last_clean":"2023-12-13T23:55:13.741390+0000","last_became_active":"2023-12-13T23:55:12.749101+0000","last_became_peered":"2023-12-13T23:55:12.749101+0000","last_unstale":"2023-12-13T23:55:13.741390+0000","last_undegraded":"2023-12-13T23:55:13.741390+0000","last_fullsized":"2023-12-13T23:55:13.741390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624137+0000","last_change":"2023-12-13T23:55:11.763071+0000","last_active":"2023-12-13T23:55:14.624137+0000","last_peered":"2023-12-13T23:55:14.624137+0000","last_clean":"2023-12-13T23:55:14.624137+0000","last_became_active":"2023-12-13T23:55:11.762774+0000","last_became_peered":"2023-12-13T23:55:11.762774+0000","last_unstale":"2023-12-13T23:55:14.624137+0000","last_undegraded":"2023-12-13T23:55:14.624137+0000","last_fullsized":"2023-12-13T23:55:14.624137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740649+0000","last_change":"2023-12-13T23:55:12.747891+0000","last_active":"2023-12-13T23:55:13.740649+0000","last_peered":"2023-12-13T23:55:13.740649+0000","last_clean":"2023-12-13T23:55:13.740649+0000","last_became_active":"2023-12-13T23:55:12.747629+0000","last_became_peered":"2023-12-13T23:55:12.747629+0000","last_unstale":"2023-12-13T23:55:13.740649+0000","last_undegraded":"2023-12-13T23:55:13.740649+0000","last_fullsized":"2023-12-13T23:55:13.740649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624263+0000","last_change":"2023-12-13T23:55:11.768277+0000","last_active":"2023-12-13T23:55:14.624263+0000","last_peered":"2023-12-13T23:55:14.624263+0000","last_clean":"2023-12-13T23:55:14.624263+0000","last_became_active":"2023-12-13T23:55:11.768039+0000","last_became_peered":"2023-12-13T23:55:11.768039+0000","last_unstale":"2023-12-13T23:55:14.624263+0000","last_undegraded":"2023-12-13T23:55:14.624263+0000","last_fullsized":"2023-12-13T23:55:14.624263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624351+0000","last_change":"2023-12-13T23:55:12.758134+0000","last_active":"2023-12-13T23:55:14.624351+0000","last_peered":"2023-12-13T23:55:14.624351+0000","last_clean":"2023-12-13T23:55:14.624351+0000","last_became_active":"2023-12-13T23:55:12.757679+0000","last_became_peered":"2023-12-13T23:55:12.757679+0000","last_unstale":"2023-12-13T23:55:14.624351+0000","last_undegraded":"2023-12-13T23:55:14.624351+0000","last_fullsized":"2023-12-13T23:55:14.624351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625036+0000","last_change":"2023-12-13T23:55:11.765748+0000","last_active":"2023-12-13T23:55:14.625036+0000","last_peered":"2023-12-13T23:55:14.625036+0000","last_clean":"2023-12-13T23:55:14.625036+0000","last_became_active":"2023-12-13T23:55:11.765320+0000","last_became_peered":"2023-12-13T23:55:11.765320+0000","last_unstale":"2023-12-13T23:55:14.625036+0000","last_undegraded":"2023-12-13T23:55:14.625036+0000","last_fullsized":"2023-12-13T23:55:14.625036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740530+0000","last_change":"2023-12-13T23:55:12.747127+0000","last_active":"2023-12-13T23:55:13.740530+0000","last_peered":"2023-12-13T23:55:13.740530+0000","last_clean":"2023-12-13T23:55:13.740530+0000","last_became_active":"2023-12-13T23:55:12.746838+0000","last_became_peered":"2023-12-13T23:55:12.746838+0000","last_unstale":"2023-12-13T23:55:13.740530+0000","last_undegraded":"2023-12-13T23:55:13.740530+0000","last_fullsized":"2023-12-13T23:55:13.740530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623961+0000","last_change":"2023-12-13T23:55:11.765098+0000","last_active":"2023-12-13T23:55:14.623961+0000","last_peered":"2023-12-13T23:55:14.623961+0000","last_clean":"2023-12-13T23:55:14.623961+0000","last_became_active":"2023-12-13T23:55:11.764746+0000","last_became_peered":"2023-12-13T23:55:11.764746+0000","last_unstale":"2023-12-13T23:55:14.623961+0000","last_undegraded":"2023-12-13T23:55:14.623961+0000","last_fullsized":"2023-12-13T23:55:14.623961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624195+0000","last_change":"2023-12-13T23:55:12.759703+0000","last_active":"2023-12-13T23:55:14.624195+0000","last_peered":"2023-12-13T23:55:14.624195+0000","last_clean":"2023-12-13T23:55:14.624195+0000","last_became_active":"2023-12-13T23:55:12.759371+0000","last_became_peered":"2023-12-13T23:55:12.759371+0000","last_unstale":"2023-12-13T23:55:14.624195+0000","last_undegraded":"2023-12-13T23:55:14.624195+0000","last_fullsized":"2023-12-13T23:55:14.624195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623693+0000","last_change":"2023-12-13T23:55:11.757065+0000","last_active":"2023-12-13T23:55:14.623693+0000","last_peered":"2023-12-13T23:55:14.623693+0000","last_clean":"2023-12-13T23:55:14.623693+0000","last_became_active":"2023-12-13T23:55:11.756732+0000","last_became_peered":"2023-12-13T23:55:11.756732+0000","last_unstale":"2023-12-13T23:55:14.623693+0000","last_undegraded":"2023-12-13T23:55:14.623693+0000","last_fullsized":"2023-12-13T23:55:14.623693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740938+0000","last_change":"2023-12-13T23:55:12.747168+0000","last_active":"2023-12-13T23:55:13.740938+0000","last_peered":"2023-12-13T23:55:13.740938+0000","last_clean":"2023-12-13T23:55:13.740938+0000","last_became_active":"2023-12-13T23:55:12.746860+0000","last_became_peered":"2023-12-13T23:55:12.746860+0000","last_unstale":"2023-12-13T23:55:13.740938+0000","last_undegraded":"2023-12-13T23:55:13.740938+0000","last_fullsized":"2023-12-13T23:55:13.740938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740851+0000","last_change":"2023-12-13T23:55:11.765922+0000","last_active":"2023-12-13T23:55:13.740851+0000","last_peered":"2023-12-13T23:55:13.740851+0000","last_clean":"2023-12-13T23:55:13.740851+0000","last_became_active":"2023-12-13T23:55:11.765641+0000","last_became_peered":"2023-12-13T23:55:11.765641+0000","last_unstale":"2023-12-13T23:55:13.740851+0000","last_undegraded":"2023-12-13T23:55:13.740851+0000","last_fullsized":"2023-12-13T23:55:13.740851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:07:21.569553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741605+0000","last_change":"2023-12-13T23:55:12.747433+0000","last_active":"2023-12-13T23:55:13.741605+0000","last_peered":"2023-12-13T23:55:13.741605+0000","last_clean":"2023-12-13T23:55:13.741605+0000","last_became_active":"2023-12-13T23:55:12.747169+0000","last_became_peered":"2023-12-13T23:55:12.747169+0000","last_unstale":"2023-12-13T23:55:13.741605+0000","last_undegraded":"2023-12-13T23:55:13.741605+0000","last_fullsized":"2023-12-13T23:55:13.741605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:03:57.776819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624072+0000","last_change":"2023-12-13T23:55:11.762985+0000","last_active":"2023-12-13T23:55:14.624072+0000","last_peered":"2023-12-13T23:55:14.624072+0000","last_clean":"2023-12-13T23:55:14.624072+0000","last_became_active":"2023-12-13T23:55:11.762654+0000","last_became_peered":"2023-12-13T23:55:11.762654+0000","last_unstale":"2023-12-13T23:55:14.624072+0000","last_undegraded":"2023-12-13T23:55:14.624072+0000","last_fullsized":"2023-12-13T23:55:14.624072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741322+0000","last_change":"2023-12-13T23:55:12.756265+0000","last_active":"2023-12-13T23:55:13.741322+0000","last_peered":"2023-12-13T23:55:13.741322+0000","last_clean":"2023-12-13T23:55:13.741322+0000","last_became_active":"2023-12-13T23:55:12.756003+0000","last_became_peered":"2023-12-13T23:55:12.756003+0000","last_unstale":"2023-12-13T23:55:13.741322+0000","last_undegraded":"2023-12-13T23:55:13.741322+0000","last_fullsized":"2023-12-13T23:55:13.741322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:26:45.885932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613709+0000","last_change":"2023-12-13T23:55:11.766945+0000","last_active":"2023-12-13T23:55:14.613709+0000","last_peered":"2023-12-13T23:55:14.613709+0000","last_clean":"2023-12-13T23:55:14.613709+0000","last_became_active":"2023-12-13T23:55:11.766671+0000","last_became_peered":"2023-12-13T23:55:11.766671+0000","last_unstale":"2023-12-13T23:55:14.613709+0000","last_undegraded":"2023-12-13T23:55:14.613709+0000","last_fullsized":"2023-12-13T23:55:14.613709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624646+0000","last_change":"2023-12-13T23:55:12.752034+0000","last_active":"2023-12-13T23:55:14.624646+0000","last_peered":"2023-12-13T23:55:14.624646+0000","last_clean":"2023-12-13T23:55:14.624646+0000","last_became_active":"2023-12-13T23:55:12.751764+0000","last_became_peered":"2023-12-13T23:55:12.751764+0000","last_unstale":"2023-12-13T23:55:14.624646+0000","last_undegraded":"2023-12-13T23:55:14.624646+0000","last_fullsized":"2023-12-13T23:55:14.624646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623912+0000","last_change":"2023-12-13T23:55:11.765879+0000","last_active":"2023-12-13T23:55:14.623912+0000","last_peered":"2023-12-13T23:55:14.623912+0000","last_clean":"2023-12-13T23:55:14.623912+0000","last_became_active":"2023-12-13T23:55:11.765607+0000","last_became_peered":"2023-12-13T23:55:11.765607+0000","last_unstale":"2023-12-13T23:55:14.623912+0000","last_undegraded":"2023-12-13T23:55:14.623912+0000","last_fullsized":"2023-12-13T23:55:14.623912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:40.262294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740784+0000","last_change":"2023-12-13T23:55:12.756349+0000","last_active":"2023-12-13T23:55:13.740784+0000","last_peered":"2023-12-13T23:55:13.740784+0000","last_clean":"2023-12-13T23:55:13.740784+0000","last_became_active":"2023-12-13T23:55:12.756033+0000","last_became_peered":"2023-12-13T23:55:12.756033+0000","last_unstale":"2023-12-13T23:55:13.740784+0000","last_undegraded":"2023-12-13T23:55:13.740784+0000","last_fullsized":"2023-12-13T23:55:13.740784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-13T23:55:12.739242+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:12.739242+0000","last_peered":"2023-12-13T23:55:12.739242+0000","last_clean":"2023-12-13T23:55:12.739242+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:12.739242+0000","last_undegraded":"2023-12-13T23:55:12.739242+0000","last_fullsized":"2023-12-13T23:55:12.739242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624508+0000","last_change":"2023-12-13T23:55:11.758971+0000","last_active":"2023-12-13T23:55:14.624508+0000","last_peered":"2023-12-13T23:55:14.624508+0000","last_clean":"2023-12-13T23:55:14.624508+0000","last_became_active":"2023-12-13T23:55:11.758594+0000","last_became_peered":"2023-12-13T23:55:11.758594+0000","last_unstale":"2023-12-13T23:55:14.624508+0000","last_undegraded":"2023-12-13T23:55:14.624508+0000","last_fullsized":"2023-12-13T23:55:14.624508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737699+0000","last_change":"2023-12-13T23:55:12.755959+0000","last_active":"2023-12-13T23:55:13.737699+0000","last_peered":"2023-12-13T23:55:13.737699+0000","last_clean":"2023-12-13T23:55:13.737699+0000","last_became_active":"2023-12-13T23:55:12.755662+0000","last_became_peered":"2023-12-13T23:55:12.755662+0000","last_unstale":"2023-12-13T23:55:13.737699+0000","last_undegraded":"2023-12-13T23:55:13.737699+0000","last_fullsized":"2023-12-13T23:55:13.737699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740316+0000","last_change":"2023-12-13T23:55:11.767937+0000","last_active":"2023-12-13T23:55:13.740316+0000","last_peered":"2023-12-13T23:55:13.740316+0000","last_clean":"2023-12-13T23:55:13.740316+0000","last_became_active":"2023-12-13T23:55:11.767613+0000","last_became_peered":"2023-12-13T23:55:11.767613+0000","last_unstale":"2023-12-13T23:55:13.740316+0000","last_undegraded":"2023-12-13T23:55:13.740316+0000","last_fullsized":"2023-12-13T23:55:13.740316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:53.188642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624112+0000","last_change":"2023-12-13T23:55:12.757755+0000","last_active":"2023-12-13T23:55:14.624112+0000","last_peered":"2023-12-13T23:55:14.624112+0000","last_clean":"2023-12-13T23:55:14.624112+0000","last_became_active":"2023-12-13T23:55:12.757456+0000","last_became_peered":"2023-12-13T23:55:12.757456+0000","last_unstale":"2023-12-13T23:55:14.624112+0000","last_undegraded":"2023-12-13T23:55:14.624112+0000","last_fullsized":"2023-12-13T23:55:14.624112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623940+0000","last_change":"2023-12-13T23:55:11.755829+0000","last_active":"2023-12-13T23:55:14.623940+0000","last_peered":"2023-12-13T23:55:14.623940+0000","last_clean":"2023-12-13T23:55:14.623940+0000","last_became_active":"2023-12-13T23:55:11.755573+0000","last_became_peered":"2023-12-13T23:55:11.755573+0000","last_unstale":"2023-12-13T23:55:14.623940+0000","last_undegraded":"2023-12-13T23:55:14.623940+0000","last_fullsized":"2023-12-13T23:55:14.623940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:56.833424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613504+0000","last_change":"2023-12-13T23:55:12.749732+0000","last_active":"2023-12-13T23:55:14.613504+0000","last_peered":"2023-12-13T23:55:14.613504+0000","last_clean":"2023-12-13T23:55:14.613504+0000","last_became_active":"2023-12-13T23:55:12.749473+0000","last_became_peered":"2023-12-13T23:55:12.749473+0000","last_unstale":"2023-12-13T23:55:14.613504+0000","last_undegraded":"2023-12-13T23:55:14.613504+0000","last_fullsized":"2023-12-13T23:55:14.613504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625041+0000","last_change":"2023-12-13T23:55:11.765970+0000","last_active":"2023-12-13T23:55:14.625041+0000","last_peered":"2023-12-13T23:55:14.625041+0000","last_clean":"2023-12-13T23:55:14.625041+0000","last_became_active":"2023-12-13T23:55:11.765629+0000","last_became_peered":"2023-12-13T23:55:11.765629+0000","last_unstale":"2023-12-13T23:55:14.625041+0000","last_undegraded":"2023-12-13T23:55:14.625041+0000","last_fullsized":"2023-12-13T23:55:14.625041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741379+0000","last_change":"2023-12-13T23:55:12.758873+0000","last_active":"2023-12-13T23:55:13.741379+0000","last_peered":"2023-12-13T23:55:13.741379+0000","last_clean":"2023-12-13T23:55:13.741379+0000","last_became_active":"2023-12-13T23:55:12.758618+0000","last_became_peered":"2023-12-13T23:55:12.758618+0000","last_unstale":"2023-12-13T23:55:13.741379+0000","last_undegraded":"2023-12-13T23:55:13.741379+0000","last_fullsized":"2023-12-13T23:55:13.741379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613597+0000","last_change":"2023-12-13T23:55:11.762944+0000","last_active":"2023-12-13T23:55:14.613597+0000","last_peered":"2023-12-13T23:55:14.613597+0000","last_clean":"2023-12-13T23:55:14.613597+0000","last_became_active":"2023-12-13T23:55:11.762691+0000","last_became_peered":"2023-12-13T23:55:11.762691+0000","last_unstale":"2023-12-13T23:55:14.613597+0000","last_undegraded":"2023-12-13T23:55:14.613597+0000","last_fullsized":"2023-12-13T23:55:14.613597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625017+0000","last_change":"2023-12-13T23:55:12.755585+0000","last_active":"2023-12-13T23:55:14.625017+0000","last_peered":"2023-12-13T23:55:14.625017+0000","last_clean":"2023-12-13T23:55:14.625017+0000","last_became_active":"2023-12-13T23:55:12.755284+0000","last_became_peered":"2023-12-13T23:55:12.755284+0000","last_unstale":"2023-12-13T23:55:14.625017+0000","last_undegraded":"2023-12-13T23:55:14.625017+0000","last_fullsized":"2023-12-13T23:55:14.625017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741015+0000","last_change":"2023-12-13T23:55:11.770291+0000","last_active":"2023-12-13T23:55:13.741015+0000","last_peered":"2023-12-13T23:55:13.741015+0000","last_clean":"2023-12-13T23:55:13.741015+0000","last_became_active":"2023-12-13T23:55:11.769672+0000","last_became_peered":"2023-12-13T23:55:11.769672+0000","last_unstale":"2023-12-13T23:55:13.741015+0000","last_undegraded":"2023-12-13T23:55:13.741015+0000","last_fullsized":"2023-12-13T23:55:13.741015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613615+0000","last_change":"2023-12-13T23:55:12.749467+0000","last_active":"2023-12-13T23:55:14.613615+0000","last_peered":"2023-12-13T23:55:14.613615+0000","last_clean":"2023-12-13T23:55:14.613615+0000","last_became_active":"2023-12-13T23:55:12.749185+0000","last_became_peered":"2023-12-13T23:55:12.749185+0000","last_unstale":"2023-12-13T23:55:14.613615+0000","last_undegraded":"2023-12-13T23:55:14.613615+0000","last_fullsized":"2023-12-13T23:55:14.613615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613740+0000","last_change":"2023-12-13T23:55:11.763643+0000","last_active":"2023-12-13T23:55:14.613740+0000","last_peered":"2023-12-13T23:55:14.613740+0000","last_clean":"2023-12-13T23:55:14.613740+0000","last_became_active":"2023-12-13T23:55:11.763378+0000","last_became_peered":"2023-12-13T23:55:11.763378+0000","last_unstale":"2023-12-13T23:55:14.613740+0000","last_undegraded":"2023-12-13T23:55:14.613740+0000","last_fullsized":"2023-12-13T23:55:14.613740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737984+0000","last_change":"2023-12-13T23:55:12.752448+0000","last_active":"2023-12-13T23:55:13.737984+0000","last_peered":"2023-12-13T23:55:13.737984+0000","last_clean":"2023-12-13T23:55:13.737984+0000","last_became_active":"2023-12-13T23:55:12.752155+0000","last_became_peered":"2023-12-13T23:55:12.752155+0000","last_unstale":"2023-12-13T23:55:13.737984+0000","last_undegraded":"2023-12-13T23:55:13.737984+0000","last_fullsized":"2023-12-13T23:55:13.737984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624790+0000","last_change":"2023-12-13T23:55:11.768156+0000","last_active":"2023-12-13T23:55:14.624790+0000","last_peered":"2023-12-13T23:55:14.624790+0000","last_clean":"2023-12-13T23:55:14.624790+0000","last_became_active":"2023-12-13T23:55:11.767878+0000","last_became_peered":"2023-12-13T23:55:11.767878+0000","last_unstale":"2023-12-13T23:55:14.624790+0000","last_undegraded":"2023-12-13T23:55:14.624790+0000","last_fullsized":"2023-12-13T23:55:14.624790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624226+0000","last_change":"2023-12-13T23:55:12.751906+0000","last_active":"2023-12-13T23:55:14.624226+0000","last_peered":"2023-12-13T23:55:14.624226+0000","last_clean":"2023-12-13T23:55:14.624226+0000","last_became_active":"2023-12-13T23:55:12.751468+0000","last_became_peered":"2023-12-13T23:55:12.751468+0000","last_unstale":"2023-12-13T23:55:14.624226+0000","last_undegraded":"2023-12-13T23:55:14.624226+0000","last_fullsized":"2023-12-13T23:55:14.624226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740225+0000","last_change":"2023-12-13T23:55:11.768417+0000","last_active":"2023-12-13T23:55:13.740225+0000","last_peered":"2023-12-13T23:55:13.740225+0000","last_clean":"2023-12-13T23:55:13.740225+0000","last_became_active":"2023-12-13T23:55:11.768141+0000","last_became_peered":"2023-12-13T23:55:11.768141+0000","last_unstale":"2023-12-13T23:55:13.740225+0000","last_undegraded":"2023-12-13T23:55:13.740225+0000","last_fullsized":"2023-12-13T23:55:13.740225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740516+0000","last_change":"2023-12-13T23:55:12.751650+0000","last_active":"2023-12-13T23:55:13.740516+0000","last_peered":"2023-12-13T23:55:13.740516+0000","last_clean":"2023-12-13T23:55:13.740516+0000","last_became_active":"2023-12-13T23:55:12.749884+0000","last_became_peered":"2023-12-13T23:55:12.749884+0000","last_unstale":"2023-12-13T23:55:13.740516+0000","last_undegraded":"2023-12-13T23:55:13.740516+0000","last_fullsized":"2023-12-13T23:55:13.740516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:12.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613607+0000","last_change":"2023-12-13T23:55:11.761362+0000","last_active":"2023-12-13T23:55:14.613607+0000","last_peered":"2023-12-13T23:55:14.613607+0000","last_clean":"2023-12-13T23:55:14.613607+0000","last_became_active":"2023-12-13T23:55:11.761091+0000","last_became_peered":"2023-12-13T23:55:11.761091+0000","last_unstale":"2023-12-13T23:55:14.613607+0000","last_undegraded":"2023-12-13T23:55:14.613607+0000","last_fullsized":"2023-12-13T23:55:14.613607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:45.292370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737621+0000","last_change":"2023-12-13T23:55:12.747019+0000","last_active":"2023-12-13T23:55:13.737621+0000","last_peered":"2023-12-13T23:55:13.737621+0000","last_clean":"2023-12-13T23:55:13.737621+0000","last_became_active":"2023-12-13T23:55:12.746720+0000","last_became_peered":"2023-12-13T23:55:12.746720+0000","last_unstale":"2023-12-13T23:55:13.737621+0000","last_undegraded":"2023-12-13T23:55:13.737621+0000","last_fullsized":"2023-12-13T23:55:13.737621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613672+0000","last_change":"2023-12-13T23:55:11.761235+0000","last_active":"2023-12-13T23:55:14.613672+0000","last_peered":"2023-12-13T23:55:14.613672+0000","last_clean":"2023-12-13T23:55:14.613672+0000","last_became_active":"2023-12-13T23:55:11.760961+0000","last_became_peered":"2023-12-13T23:55:11.760961+0000","last_unstale":"2023-12-13T23:55:14.613672+0000","last_undegraded":"2023-12-13T23:55:14.613672+0000","last_fullsized":"2023-12-13T23:55:14.613672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737873+0000","last_change":"2023-12-13T23:55:12.757945+0000","last_active":"2023-12-13T23:55:13.737873+0000","last_peered":"2023-12-13T23:55:13.737873+0000","last_clean":"2023-12-13T23:55:13.737873+0000","last_became_active":"2023-12-13T23:55:12.757685+0000","last_became_peered":"2023-12-13T23:55:12.757685+0000","last_unstale":"2023-12-13T23:55:13.737873+0000","last_undegraded":"2023-12-13T23:55:13.737873+0000","last_fullsized":"2023-12-13T23:55:13.737873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740608+0000","last_change":"2023-12-13T23:55:11.769857+0000","last_active":"2023-12-13T23:55:13.740608+0000","last_peered":"2023-12-13T23:55:13.740608+0000","last_clean":"2023-12-13T23:55:13.740608+0000","last_became_active":"2023-12-13T23:55:11.769503+0000","last_became_peered":"2023-12-13T23:55:11.769503+0000","last_unstale":"2023-12-13T23:55:13.740608+0000","last_undegraded":"2023-12-13T23:55:13.740608+0000","last_fullsized":"2023-12-13T23:55:13.740608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624730+0000","last_change":"2023-12-13T23:55:12.747991+0000","last_active":"2023-12-13T23:55:14.624730+0000","last_peered":"2023-12-13T23:55:14.624730+0000","last_clean":"2023-12-13T23:55:14.624730+0000","last_became_active":"2023-12-13T23:55:12.746787+0000","last_became_peered":"2023-12-13T23:55:12.746787+0000","last_unstale":"2023-12-13T23:55:14.624730+0000","last_undegraded":"2023-12-13T23:55:14.624730+0000","last_fullsized":"2023-12-13T23:55:14.624730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624966+0000","last_change":"2023-12-13T23:55:12.755883+0000","last_active":"2023-12-13T23:55:14.624966+0000","last_peered":"2023-12-13T23:55:14.624966+0000","last_clean":"2023-12-13T23:55:14.624966+0000","last_became_active":"2023-12-13T23:55:12.755573+0000","last_became_peered":"2023-12-13T23:55:12.755573+0000","last_unstale":"2023-12-13T23:55:14.624966+0000","last_undegraded":"2023-12-13T23:55:14.624966+0000","last_fullsized":"2023-12-13T23:55:14.624966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741033+0000","last_change":"2023-12-13T23:55:11.769532+0000","last_active":"2023-12-13T23:55:13.741033+0000","last_peered":"2023-12-13T23:55:13.741033+0000","last_clean":"2023-12-13T23:55:13.741033+0000","last_became_active":"2023-12-13T23:55:11.769251+0000","last_became_peered":"2023-12-13T23:55:11.769251+0000","last_unstale":"2023-12-13T23:55:13.741033+0000","last_undegraded":"2023-12-13T23:55:13.741033+0000","last_fullsized":"2023-12-13T23:55:13.741033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624446+0000","last_change":"2023-12-13T23:55:12.757891+0000","last_active":"2023-12-13T23:55:14.624446+0000","last_peered":"2023-12-13T23:55:14.624446+0000","last_clean":"2023-12-13T23:55:14.624446+0000","last_became_active":"2023-12-13T23:55:12.757643+0000","last_became_peered":"2023-12-13T23:55:12.757643+0000","last_unstale":"2023-12-13T23:55:14.624446+0000","last_undegraded":"2023-12-13T23:55:14.624446+0000","last_fullsized":"2023-12-13T23:55:14.624446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740911+0000","last_change":"2023-12-13T23:55:11.769811+0000","last_active":"2023-12-13T23:55:13.740911+0000","last_peered":"2023-12-13T23:55:13.740911+0000","last_clean":"2023-12-13T23:55:13.740911+0000","last_became_active":"2023-12-13T23:55:11.769483+0000","last_became_peered":"2023-12-13T23:55:11.769483+0000","last_unstale":"2023-12-13T23:55:13.740911+0000","last_undegraded":"2023-12-13T23:55:13.740911+0000","last_fullsized":"2023-12-13T23:55:13.740911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624275+0000","last_change":"2023-12-13T23:55:12.758511+0000","last_active":"2023-12-13T23:55:14.624275+0000","last_peered":"2023-12-13T23:55:14.624275+0000","last_clean":"2023-12-13T23:55:14.624275+0000","last_became_active":"2023-12-13T23:55:12.757824+0000","last_became_peered":"2023-12-13T23:55:12.757824+0000","last_unstale":"2023-12-13T23:55:14.624275+0000","last_undegraded":"2023-12-13T23:55:14.624275+0000","last_fullsized":"2023-12-13T23:55:14.624275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737890+0000","last_change":"2023-12-13T23:55:11.761808+0000","last_active":"2023-12-13T23:55:13.737890+0000","last_peered":"2023-12-13T23:55:13.737890+0000","last_clean":"2023-12-13T23:55:13.737890+0000","last_became_active":"2023-12-13T23:55:11.761559+0000","last_became_peered":"2023-12-13T23:55:11.761559+0000","last_unstale":"2023-12-13T23:55:13.737890+0000","last_undegraded":"2023-12-13T23:55:13.737890+0000","last_fullsized":"2023-12-13T23:55:13.737890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737814+0000","last_change":"2023-12-13T23:55:12.748462+0000","last_active":"2023-12-13T23:55:13.737814+0000","last_peered":"2023-12-13T23:55:13.737814+0000","last_clean":"2023-12-13T23:55:13.737814+0000","last_became_active":"2023-12-13T23:55:12.748166+0000","last_became_peered":"2023-12-13T23:55:12.748166+0000","last_unstale":"2023-12-13T23:55:13.737814+0000","last_undegraded":"2023-12-13T23:55:13.737814+0000","last_fullsized":"2023-12-13T23:55:13.737814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624331+0000","last_change":"2023-12-13T23:55:11.771111+0000","last_active":"2023-12-13T23:55:14.624331+0000","last_peered":"2023-12-13T23:55:14.624331+0000","last_clean":"2023-12-13T23:55:14.624331+0000","last_became_active":"2023-12-13T23:55:11.770795+0000","last_became_peered":"2023-12-13T23:55:11.770795+0000","last_unstale":"2023-12-13T23:55:14.624331+0000","last_undegraded":"2023-12-13T23:55:14.624331+0000","last_fullsized":"2023-12-13T23:55:14.624331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:17.761 INFO:tasks.ceph.mds.d.smithi203.stderr:2023-12-13T23:55:17.763+0000 7f515f001640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:55:20.070 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:20.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.181+0000 7f33e1200640 1 Processor -- start 2023-12-13T23:55:20.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.181+0000 7f33e1200640 1 -- start start 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33e1200640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f33dc1609f0 0x7f33dc0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33e1200640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33e1200640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33dc160dd0 con 0x7f33dc06c290 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33e1200640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33dc06c710 con 0x7f33dc1609f0 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33e1200640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0d0320 con 0x7f33dc06b910 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f33dc1609f0 0x7f33dc0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f33dc1609f0 0x7f33dc0cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48758/0 (socket says 172.21.15.124:48758) 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 -- 172.21.15.124:0/722451158 learned_addr learned my addr 172.21.15.124:0/722451158 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33d3fff640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 -- 172.21.15.124:0/722451158 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 msgr2=0x7f33dc06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc06bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 -- 172.21.15.124:0/722451158 >> v1:172.21.15.203:6789/0 conn(0x7f33dc06c290 legacy=0x7f33dc0cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.182+0000 7f33dad9d640 1 -- 172.21.15.124:0/722451158 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc0d0600 con 0x7f33dc1609f0 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33dad9d640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc1609f0 0x7f33dc0cfad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47846ebbf0f3864d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33da59c640 1 -- 172.21.15.124:0/722451158 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031640 con 0x7f33dc1609f0 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33da59c640 1 -- 172.21.15.124:0/722451158 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc0317a0 con 0x7f33dc1609f0 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33da59c640 1 -- 172.21.15.124:0/722451158 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33cc031b30 con 0x7f33dc1609f0 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33e1200640 1 -- 172.21.15.124:0/722451158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc1609f0 msgr2=0x7f33dc0cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33e1200640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc1609f0 0x7f33dc0cfad0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009870 tx=0x7f33cc02b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33e1200640 1 -- 172.21.15.124:0/722451158 shutdown_connections 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33e1200640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.183+0000 7f33e1200640 1 --2- 172.21.15.124:0/722451158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc1609f0 0x7f33dc0cfad0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.184+0000 7f33e1200640 1 -- 172.21.15.124:0/722451158 >> 172.21.15.124:0/722451158 conn(0x7f33dc0d0d80 msgr2=0x7f33dc0d1190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.184+0000 7f33e1200640 1 -- 172.21.15.124:0/722451158 shutdown_connections 2023-12-13T23:55:20.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.184+0000 7f33e1200640 1 -- 172.21.15.124:0/722451158 wait complete. 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.185+0000 7f33e1200640 1 Processor -- start 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.185+0000 7f33e1200640 1 -- start start 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc0c9630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc1609f0 0x7f33dc14a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0ca8c0 con 0x7f33dc06c290 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc14ddf0 con 0x7f33dc1609f0 2023-12-13T23:55:20.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0c9cb0 con 0x7f33dc06b910 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48768/0 (socket says 172.21.15.124:48768) 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 -- 172.21.15.124:0/928336459 learned_addr learned my addr 172.21.15.124:0/928336459 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33dad9d640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc0c9630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33db59e640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc1609f0 0x7f33dc14a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 -- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 msgr2=0x7f33dc0c9630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc0c9630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 -- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc1609f0 msgr2=0x7f33dc14a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc1609f0 0x7f33dc14a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 -- 172.21.15.124:0/928336459 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0094d0 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d3fff640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c7f0 tx=0x7f33c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4004230 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33c4004390 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc0c9f90 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.186+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33c4007860 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.187+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc0ca550 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.188+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc0caa70 con 0x7f33dc06c290 2023-12-13T23:55:20.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.188+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f33c4007bd0 con 0x7f33dc06c290 2023-12-13T23:55:20.189 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.188+0000 7f33d37fe640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 0x7f33bc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.190 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.188+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33c4089a00 con 0x7f33dc06c290 2023-12-13T23:55:20.190 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.188+0000 7f33dad9d640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 0x7f33bc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.190 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.189+0000 7f33dad9d640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 0x7f33bc06d680 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.190 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.193+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f33dc0caa70 con 0x7f33dc06c290 2023-12-13T23:55:20.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.358+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33dc0cb990 con 0x7f33bc06b1c0 2023-12-13T23:55:20.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.365+0000 7f33d37fe640 1 -- 172.21.15.124:0/928336459 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264701 (secure 0 0 0) 0x7f33cc031ae0 con 0x7f33bc06b1c0 2023-12-13T23:55:20.366 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:20.367 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 msgr2=0x7f33bc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 0x7f33bc06d680 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f33cc036000 tx=0x7f33cc03c040 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 msgr2=0x7f33dc14d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f33c400c7f0 tx=0x7f33c400ccb0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 shutdown_connections 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33bc06b1c0 0x7f33bc06d680 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f33dc1609f0 0x7f33dc14a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f33dc06c290 0x7f33dc14d8b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 --2- 172.21.15.124:0/928336459 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f33dc06b910 0x7f33dc0c9630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 >> 172.21.15.124:0/928336459 conn(0x7f33dc0d0d80 msgr2=0x7f33dc0cdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:20.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 shutdown_connections 2023-12-13T23:55:20.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.368+0000 7f33e1200640 1 -- 172.21.15.124:0/928336459 wait complete. 2023-12-13T23:55:20.378 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-12-13T23:55:20.111102+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":172,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219028,"kb_used_data":2860,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755692,"statfs":{"total":773094113280,"available":772869828608,"internally_reserved":0,"allocated":2928640,"data_stored":1758704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001944"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741160+0000","last_change":"2023-12-13T23:55:12.749065+0000","last_active":"2023-12-13T23:55:13.741160+0000","last_peered":"2023-12-13T23:55:13.741160+0000","last_clean":"2023-12-13T23:55:13.741160+0000","last_became_active":"2023-12-13T23:55:12.748399+0000","last_became_peered":"2023-12-13T23:55:12.748399+0000","last_unstale":"2023-12-13T23:55:13.741160+0000","last_undegraded":"2023-12-13T23:55:13.741160+0000","last_fullsized":"2023-12-13T23:55:13.741160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613725+0000","last_change":"2023-12-13T23:55:12.751938+0000","last_active":"2023-12-13T23:55:14.613725+0000","last_peered":"2023-12-13T23:55:14.613725+0000","last_clean":"2023-12-13T23:55:14.613725+0000","last_became_active":"2023-12-13T23:55:12.751688+0000","last_became_peered":"2023-12-13T23:55:12.751688+0000","last_unstale":"2023-12-13T23:55:14.613725+0000","last_undegraded":"2023-12-13T23:55:14.613725+0000","last_fullsized":"2023-12-13T23:55:14.613725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624101+0000","last_change":"2023-12-13T23:55:12.757831+0000","last_active":"2023-12-13T23:55:14.624101+0000","last_peered":"2023-12-13T23:55:14.624101+0000","last_clean":"2023-12-13T23:55:14.624101+0000","last_became_active":"2023-12-13T23:55:12.757403+0000","last_became_peered":"2023-12-13T23:55:12.757403+0000","last_unstale":"2023-12-13T23:55:14.624101+0000","last_undegraded":"2023-12-13T23:55:14.624101+0000","last_fullsized":"2023-12-13T23:55:14.624101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624401+0000","last_change":"2023-12-13T23:55:12.757838+0000","last_active":"2023-12-13T23:55:14.624401+0000","last_peered":"2023-12-13T23:55:14.624401+0000","last_clean":"2023-12-13T23:55:14.624401+0000","last_became_active":"2023-12-13T23:55:12.757590+0000","last_became_peered":"2023-12-13T23:55:12.757590+0000","last_unstale":"2023-12-13T23:55:14.624401+0000","last_undegraded":"2023-12-13T23:55:14.624401+0000","last_fullsized":"2023-12-13T23:55:14.624401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740506+0000","last_change":"2023-12-13T23:55:12.750174+0000","last_active":"2023-12-13T23:55:13.740506+0000","last_peered":"2023-12-13T23:55:13.740506+0000","last_clean":"2023-12-13T23:55:13.740506+0000","last_became_active":"2023-12-13T23:55:12.749937+0000","last_became_peered":"2023-12-13T23:55:12.749937+0000","last_unstale":"2023-12-13T23:55:13.740506+0000","last_undegraded":"2023-12-13T23:55:13.740506+0000","last_fullsized":"2023-12-13T23:55:13.740506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624775+0000","last_change":"2023-12-13T23:55:12.760175+0000","last_active":"2023-12-13T23:55:14.624775+0000","last_peered":"2023-12-13T23:55:14.624775+0000","last_clean":"2023-12-13T23:55:14.624775+0000","last_became_active":"2023-12-13T23:55:12.759888+0000","last_became_peered":"2023-12-13T23:55:12.759888+0000","last_unstale":"2023-12-13T23:55:14.624775+0000","last_undegraded":"2023-12-13T23:55:14.624775+0000","last_fullsized":"2023-12-13T23:55:14.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740922+0000","last_change":"2023-12-13T23:55:12.758545+0000","last_active":"2023-12-13T23:55:13.740922+0000","last_peered":"2023-12-13T23:55:13.740922+0000","last_clean":"2023-12-13T23:55:13.740922+0000","last_became_active":"2023-12-13T23:55:12.758292+0000","last_became_peered":"2023-12-13T23:55:12.758292+0000","last_unstale":"2023-12-13T23:55:13.740922+0000","last_undegraded":"2023-12-13T23:55:13.740922+0000","last_fullsized":"2023-12-13T23:55:13.740922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737487+0000","last_change":"2023-12-13T23:55:12.756050+0000","last_active":"2023-12-13T23:55:13.737487+0000","last_peered":"2023-12-13T23:55:13.737487+0000","last_clean":"2023-12-13T23:55:13.737487+0000","last_became_active":"2023-12-13T23:55:12.755765+0000","last_became_peered":"2023-12-13T23:55:12.755765+0000","last_unstale":"2023-12-13T23:55:13.737487+0000","last_undegraded":"2023-12-13T23:55:13.737487+0000","last_fullsized":"2023-12-13T23:55:13.737487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741293+0000","last_change":"2023-12-13T23:55:12.751022+0000","last_active":"2023-12-13T23:55:13.741293+0000","last_peered":"2023-12-13T23:55:13.741293+0000","last_clean":"2023-12-13T23:55:13.741293+0000","last_became_active":"2023-12-13T23:55:12.750750+0000","last_became_peered":"2023-12-13T23:55:12.750750+0000","last_unstale":"2023-12-13T23:55:13.741293+0000","last_undegraded":"2023-12-13T23:55:13.741293+0000","last_fullsized":"2023-12-13T23:55:13.741293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624534+0000","last_change":"2023-12-13T23:55:12.754201+0000","last_active":"2023-12-13T23:55:14.624534+0000","last_peered":"2023-12-13T23:55:14.624534+0000","last_clean":"2023-12-13T23:55:14.624534+0000","last_became_active":"2023-12-13T23:55:12.753905+0000","last_became_peered":"2023-12-13T23:55:12.753905+0000","last_unstale":"2023-12-13T23:55:14.624534+0000","last_undegraded":"2023-12-13T23:55:14.624534+0000","last_fullsized":"2023-12-13T23:55:14.624534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624445+0000","last_change":"2023-12-13T23:55:12.756766+0000","last_active":"2023-12-13T23:55:14.624445+0000","last_peered":"2023-12-13T23:55:14.624445+0000","last_clean":"2023-12-13T23:55:14.624445+0000","last_became_active":"2023-12-13T23:55:12.756500+0000","last_became_peered":"2023-12-13T23:55:12.756500+0000","last_unstale":"2023-12-13T23:55:14.624445+0000","last_undegraded":"2023-12-13T23:55:14.624445+0000","last_fullsized":"2023-12-13T23:55:14.624445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741255+0000","last_change":"2023-12-13T23:55:12.756454+0000","last_active":"2023-12-13T23:55:13.741255+0000","last_peered":"2023-12-13T23:55:13.741255+0000","last_clean":"2023-12-13T23:55:13.741255+0000","last_became_active":"2023-12-13T23:55:12.756151+0000","last_became_peered":"2023-12-13T23:55:12.756151+0000","last_unstale":"2023-12-13T23:55:13.741255+0000","last_undegraded":"2023-12-13T23:55:13.741255+0000","last_fullsized":"2023-12-13T23:55:13.741255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:41.266746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737767+0000","last_change":"2023-12-13T23:55:12.751798+0000","last_active":"2023-12-13T23:55:13.737767+0000","last_peered":"2023-12-13T23:55:13.737767+0000","last_clean":"2023-12-13T23:55:13.737767+0000","last_became_active":"2023-12-13T23:55:12.751513+0000","last_became_peered":"2023-12-13T23:55:12.751513+0000","last_unstale":"2023-12-13T23:55:13.737767+0000","last_undegraded":"2023-12-13T23:55:13.737767+0000","last_fullsized":"2023-12-13T23:55:13.737767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624265+0000","last_change":"2023-12-13T23:55:12.757612+0000","last_active":"2023-12-13T23:55:14.624265+0000","last_peered":"2023-12-13T23:55:14.624265+0000","last_clean":"2023-12-13T23:55:14.624265+0000","last_became_active":"2023-12-13T23:55:12.757320+0000","last_became_peered":"2023-12-13T23:55:12.757320+0000","last_unstale":"2023-12-13T23:55:14.624265+0000","last_undegraded":"2023-12-13T23:55:14.624265+0000","last_fullsized":"2023-12-13T23:55:14.624265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613523+0000","last_change":"2023-12-13T23:55:12.743417+0000","last_active":"2023-12-13T23:55:14.613523+0000","last_peered":"2023-12-13T23:55:14.613523+0000","last_clean":"2023-12-13T23:55:14.613523+0000","last_became_active":"2023-12-13T23:55:12.743121+0000","last_became_peered":"2023-12-13T23:55:12.743121+0000","last_unstale":"2023-12-13T23:55:14.613523+0000","last_undegraded":"2023-12-13T23:55:14.613523+0000","last_fullsized":"2023-12-13T23:55:14.613523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624106+0000","last_change":"2023-12-13T23:55:12.761124+0000","last_active":"2023-12-13T23:55:14.624106+0000","last_peered":"2023-12-13T23:55:14.624106+0000","last_clean":"2023-12-13T23:55:14.624106+0000","last_became_active":"2023-12-13T23:55:12.760661+0000","last_became_peered":"2023-12-13T23:55:12.760661+0000","last_unstale":"2023-12-13T23:55:14.624106+0000","last_undegraded":"2023-12-13T23:55:14.624106+0000","last_fullsized":"2023-12-13T23:55:14.624106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624905+0000","last_change":"2023-12-13T23:55:12.752470+0000","last_active":"2023-12-13T23:55:14.624905+0000","last_peered":"2023-12-13T23:55:14.624905+0000","last_clean":"2023-12-13T23:55:14.624905+0000","last_became_active":"2023-12-13T23:55:12.752197+0000","last_became_peered":"2023-12-13T23:55:12.752197+0000","last_unstale":"2023-12-13T23:55:14.624905+0000","last_undegraded":"2023-12-13T23:55:14.624905+0000","last_fullsized":"2023-12-13T23:55:14.624905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623990+0000","last_change":"2023-12-13T23:55:12.748844+0000","last_active":"2023-12-13T23:55:14.623990+0000","last_peered":"2023-12-13T23:55:14.623990+0000","last_clean":"2023-12-13T23:55:14.623990+0000","last_became_active":"2023-12-13T23:55:12.748564+0000","last_became_peered":"2023-12-13T23:55:12.748564+0000","last_unstale":"2023-12-13T23:55:14.623990+0000","last_undegraded":"2023-12-13T23:55:14.623990+0000","last_fullsized":"2023-12-13T23:55:14.623990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741142+0000","last_change":"2023-12-13T23:55:12.749584+0000","last_active":"2023-12-13T23:55:13.741142+0000","last_peered":"2023-12-13T23:55:13.741142+0000","last_clean":"2023-12-13T23:55:13.741142+0000","last_became_active":"2023-12-13T23:55:12.749330+0000","last_became_peered":"2023-12-13T23:55:12.749330+0000","last_unstale":"2023-12-13T23:55:13.741142+0000","last_undegraded":"2023-12-13T23:55:13.741142+0000","last_fullsized":"2023-12-13T23:55:13.741142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624156+0000","last_change":"2023-12-13T23:55:12.746390+0000","last_active":"2023-12-13T23:55:14.624156+0000","last_peered":"2023-12-13T23:55:14.624156+0000","last_clean":"2023-12-13T23:55:14.624156+0000","last_became_active":"2023-12-13T23:55:12.746135+0000","last_became_peered":"2023-12-13T23:55:12.746135+0000","last_unstale":"2023-12-13T23:55:14.624156+0000","last_undegraded":"2023-12-13T23:55:14.624156+0000","last_fullsized":"2023-12-13T23:55:14.624156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624034+0000","last_change":"2023-12-13T23:55:12.752805+0000","last_active":"2023-12-13T23:55:14.624034+0000","last_peered":"2023-12-13T23:55:14.624034+0000","last_clean":"2023-12-13T23:55:14.624034+0000","last_became_active":"2023-12-13T23:55:12.752447+0000","last_became_peered":"2023-12-13T23:55:12.752447+0000","last_unstale":"2023-12-13T23:55:14.624034+0000","last_undegraded":"2023-12-13T23:55:14.624034+0000","last_fullsized":"2023-12-13T23:55:14.624034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:24.291337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624917+0000","last_change":"2023-12-13T23:55:12.754499+0000","last_active":"2023-12-13T23:55:14.624917+0000","last_peered":"2023-12-13T23:55:14.624917+0000","last_clean":"2023-12-13T23:55:14.624917+0000","last_became_active":"2023-12-13T23:55:12.753996+0000","last_became_peered":"2023-12-13T23:55:12.753996+0000","last_unstale":"2023-12-13T23:55:14.624917+0000","last_undegraded":"2023-12-13T23:55:14.624917+0000","last_fullsized":"2023-12-13T23:55:14.624917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737534+0000","last_change":"2023-12-13T23:55:12.750598+0000","last_active":"2023-12-13T23:55:13.737534+0000","last_peered":"2023-12-13T23:55:13.737534+0000","last_clean":"2023-12-13T23:55:13.737534+0000","last_became_active":"2023-12-13T23:55:12.750296+0000","last_became_peered":"2023-12-13T23:55:12.750296+0000","last_unstale":"2023-12-13T23:55:13.737534+0000","last_undegraded":"2023-12-13T23:55:13.737534+0000","last_fullsized":"2023-12-13T23:55:13.737534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624327+0000","last_change":"2023-12-13T23:55:12.750435+0000","last_active":"2023-12-13T23:55:14.624327+0000","last_peered":"2023-12-13T23:55:14.624327+0000","last_clean":"2023-12-13T23:55:14.624327+0000","last_became_active":"2023-12-13T23:55:12.750151+0000","last_became_peered":"2023-12-13T23:55:12.750151+0000","last_unstale":"2023-12-13T23:55:14.624327+0000","last_undegraded":"2023-12-13T23:55:14.624327+0000","last_fullsized":"2023-12-13T23:55:14.624327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737900+0000","last_change":"2023-12-13T23:55:12.750495+0000","last_active":"2023-12-13T23:55:13.737900+0000","last_peered":"2023-12-13T23:55:13.737900+0000","last_clean":"2023-12-13T23:55:13.737900+0000","last_became_active":"2023-12-13T23:55:12.750227+0000","last_became_peered":"2023-12-13T23:55:12.750227+0000","last_unstale":"2023-12-13T23:55:13.737900+0000","last_undegraded":"2023-12-13T23:55:13.737900+0000","last_fullsized":"2023-12-13T23:55:13.737900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624667+0000","last_change":"2023-12-13T23:55:12.751349+0000","last_active":"2023-12-13T23:55:14.624667+0000","last_peered":"2023-12-13T23:55:14.624667+0000","last_clean":"2023-12-13T23:55:14.624667+0000","last_became_active":"2023-12-13T23:55:12.751057+0000","last_became_peered":"2023-12-13T23:55:12.751057+0000","last_unstale":"2023-12-13T23:55:14.624667+0000","last_undegraded":"2023-12-13T23:55:14.624667+0000","last_fullsized":"2023-12-13T23:55:14.624667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613647+0000","last_change":"2023-12-13T23:55:12.748079+0000","last_active":"2023-12-13T23:55:14.613647+0000","last_peered":"2023-12-13T23:55:14.613647+0000","last_clean":"2023-12-13T23:55:14.613647+0000","last_became_active":"2023-12-13T23:55:12.747722+0000","last_became_peered":"2023-12-13T23:55:12.747722+0000","last_unstale":"2023-12-13T23:55:14.613647+0000","last_undegraded":"2023-12-13T23:55:14.613647+0000","last_fullsized":"2023-12-13T23:55:14.613647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737802+0000","last_change":"2023-12-13T23:55:12.747671+0000","last_active":"2023-12-13T23:55:13.737802+0000","last_peered":"2023-12-13T23:55:13.737802+0000","last_clean":"2023-12-13T23:55:13.737802+0000","last_became_active":"2023-12-13T23:55:12.747431+0000","last_became_peered":"2023-12-13T23:55:12.747431+0000","last_unstale":"2023-12-13T23:55:13.737802+0000","last_undegraded":"2023-12-13T23:55:13.737802+0000","last_fullsized":"2023-12-13T23:55:13.737802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624186+0000","last_change":"2023-12-13T23:55:12.751994+0000","last_active":"2023-12-13T23:55:14.624186+0000","last_peered":"2023-12-13T23:55:14.624186+0000","last_clean":"2023-12-13T23:55:14.624186+0000","last_became_active":"2023-12-13T23:55:12.751598+0000","last_became_peered":"2023-12-13T23:55:12.751598+0000","last_unstale":"2023-12-13T23:55:14.624186+0000","last_undegraded":"2023-12-13T23:55:14.624186+0000","last_fullsized":"2023-12-13T23:55:14.624186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624938+0000","last_change":"2023-12-13T23:55:12.754312+0000","last_active":"2023-12-13T23:55:14.624938+0000","last_peered":"2023-12-13T23:55:14.624938+0000","last_clean":"2023-12-13T23:55:14.624938+0000","last_became_active":"2023-12-13T23:55:12.753618+0000","last_became_peered":"2023-12-13T23:55:12.753618+0000","last_unstale":"2023-12-13T23:55:14.624938+0000","last_undegraded":"2023-12-13T23:55:14.624938+0000","last_fullsized":"2023-12-13T23:55:14.624938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623822+0000","last_change":"2023-12-13T23:55:12.755004+0000","last_active":"2023-12-13T23:55:14.623822+0000","last_peered":"2023-12-13T23:55:14.623822+0000","last_clean":"2023-12-13T23:55:14.623822+0000","last_became_active":"2023-12-13T23:55:12.754761+0000","last_became_peered":"2023-12-13T23:55:12.754761+0000","last_unstale":"2023-12-13T23:55:14.623822+0000","last_undegraded":"2023-12-13T23:55:14.623822+0000","last_fullsized":"2023-12-13T23:55:14.623822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624213+0000","last_change":"2023-12-13T23:55:12.758828+0000","last_active":"2023-12-13T23:55:14.624213+0000","last_peered":"2023-12-13T23:55:14.624213+0000","last_clean":"2023-12-13T23:55:14.624213+0000","last_became_active":"2023-12-13T23:55:12.758520+0000","last_became_peered":"2023-12-13T23:55:12.758520+0000","last_unstale":"2023-12-13T23:55:14.624213+0000","last_undegraded":"2023-12-13T23:55:14.624213+0000","last_fullsized":"2023-12-13T23:55:14.624213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740370+0000","last_change":"2023-12-13T23:55:12.745882+0000","last_active":"2023-12-13T23:55:13.740370+0000","last_peered":"2023-12-13T23:55:13.740370+0000","last_clean":"2023-12-13T23:55:13.740370+0000","last_became_active":"2023-12-13T23:55:12.744988+0000","last_became_peered":"2023-12-13T23:55:12.744988+0000","last_unstale":"2023-12-13T23:55:13.740370+0000","last_undegraded":"2023-12-13T23:55:13.740370+0000","last_fullsized":"2023-12-13T23:55:13.740370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:57.837875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737851+0000","last_change":"2023-12-13T23:55:11.769713+0000","last_active":"2023-12-13T23:55:13.737851+0000","last_peered":"2023-12-13T23:55:13.737851+0000","last_clean":"2023-12-13T23:55:13.737851+0000","last_became_active":"2023-12-13T23:55:11.769540+0000","last_became_peered":"2023-12-13T23:55:11.769540+0000","last_unstale":"2023-12-13T23:55:13.737851+0000","last_undegraded":"2023-12-13T23:55:13.737851+0000","last_fullsized":"2023-12-13T23:55:13.737851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624177+0000","last_change":"2023-12-13T23:55:12.752916+0000","last_active":"2023-12-13T23:55:14.624177+0000","last_peered":"2023-12-13T23:55:14.624177+0000","last_clean":"2023-12-13T23:55:14.624177+0000","last_became_active":"2023-12-13T23:55:12.751877+0000","last_became_peered":"2023-12-13T23:55:12.751877+0000","last_unstale":"2023-12-13T23:55:14.624177+0000","last_undegraded":"2023-12-13T23:55:14.624177+0000","last_fullsized":"2023-12-13T23:55:14.624177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624064+0000","last_change":"2023-12-13T23:55:11.767168+0000","last_active":"2023-12-13T23:55:14.624064+0000","last_peered":"2023-12-13T23:55:14.624064+0000","last_clean":"2023-12-13T23:55:14.624064+0000","last_became_active":"2023-12-13T23:55:11.766944+0000","last_became_peered":"2023-12-13T23:55:11.766944+0000","last_unstale":"2023-12-13T23:55:14.624064+0000","last_undegraded":"2023-12-13T23:55:14.624064+0000","last_fullsized":"2023-12-13T23:55:14.624064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737587+0000","last_change":"2023-12-13T23:55:11.764302+0000","last_active":"2023-12-13T23:55:13.737587+0000","last_peered":"2023-12-13T23:55:13.737587+0000","last_clean":"2023-12-13T23:55:13.737587+0000","last_became_active":"2023-12-13T23:55:11.763930+0000","last_became_peered":"2023-12-13T23:55:11.763930+0000","last_unstale":"2023-12-13T23:55:13.737587+0000","last_undegraded":"2023-12-13T23:55:13.737587+0000","last_fullsized":"2023-12-13T23:55:13.737587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737560+0000","last_change":"2023-12-13T23:55:12.760219+0000","last_active":"2023-12-13T23:55:13.737560+0000","last_peered":"2023-12-13T23:55:13.737560+0000","last_clean":"2023-12-13T23:55:13.737560+0000","last_became_active":"2023-12-13T23:55:12.759890+0000","last_became_peered":"2023-12-13T23:55:12.759890+0000","last_unstale":"2023-12-13T23:55:13.737560+0000","last_undegraded":"2023-12-13T23:55:13.737560+0000","last_fullsized":"2023-12-13T23:55:13.737560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624404+0000","last_change":"2023-12-13T23:55:11.763128+0000","last_active":"2023-12-13T23:55:14.624404+0000","last_peered":"2023-12-13T23:55:14.624404+0000","last_clean":"2023-12-13T23:55:14.624404+0000","last_became_active":"2023-12-13T23:55:11.762848+0000","last_became_peered":"2023-12-13T23:55:11.762848+0000","last_unstale":"2023-12-13T23:55:14.624404+0000","last_undegraded":"2023-12-13T23:55:14.624404+0000","last_fullsized":"2023-12-13T23:55:14.624404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613510+0000","last_change":"2023-12-13T23:55:12.747367+0000","last_active":"2023-12-13T23:55:14.613510+0000","last_peered":"2023-12-13T23:55:14.613510+0000","last_clean":"2023-12-13T23:55:14.613510+0000","last_became_active":"2023-12-13T23:55:12.747074+0000","last_became_peered":"2023-12-13T23:55:12.747074+0000","last_unstale":"2023-12-13T23:55:14.613510+0000","last_undegraded":"2023-12-13T23:55:14.613510+0000","last_fullsized":"2023-12-13T23:55:14.613510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624348+0000","last_change":"2023-12-13T23:55:11.766140+0000","last_active":"2023-12-13T23:55:14.624348+0000","last_peered":"2023-12-13T23:55:14.624348+0000","last_clean":"2023-12-13T23:55:14.624348+0000","last_became_active":"2023-12-13T23:55:11.765848+0000","last_became_peered":"2023-12-13T23:55:11.765848+0000","last_unstale":"2023-12-13T23:55:14.624348+0000","last_undegraded":"2023-12-13T23:55:14.624348+0000","last_fullsized":"2023-12-13T23:55:14.624348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624786+0000","last_change":"2023-12-13T23:55:12.750678+0000","last_active":"2023-12-13T23:55:14.624786+0000","last_peered":"2023-12-13T23:55:14.624786+0000","last_clean":"2023-12-13T23:55:14.624786+0000","last_became_active":"2023-12-13T23:55:12.750200+0000","last_became_peered":"2023-12-13T23:55:12.750200+0000","last_unstale":"2023-12-13T23:55:14.624786+0000","last_undegraded":"2023-12-13T23:55:14.624786+0000","last_fullsized":"2023-12-13T23:55:14.624786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624863+0000","last_change":"2023-12-13T23:55:11.763844+0000","last_active":"2023-12-13T23:55:14.624863+0000","last_peered":"2023-12-13T23:55:14.624863+0000","last_clean":"2023-12-13T23:55:14.624863+0000","last_became_active":"2023-12-13T23:55:11.763574+0000","last_became_peered":"2023-12-13T23:55:11.763574+0000","last_unstale":"2023-12-13T23:55:14.624863+0000","last_undegraded":"2023-12-13T23:55:14.624863+0000","last_fullsized":"2023-12-13T23:55:14.624863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:41:34.696890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624340+0000","last_change":"2023-12-13T23:55:12.754675+0000","last_active":"2023-12-13T23:55:14.624340+0000","last_peered":"2023-12-13T23:55:14.624340+0000","last_clean":"2023-12-13T23:55:14.624340+0000","last_became_active":"2023-12-13T23:55:12.753740+0000","last_became_peered":"2023-12-13T23:55:12.753740+0000","last_unstale":"2023-12-13T23:55:14.624340+0000","last_undegraded":"2023-12-13T23:55:14.624340+0000","last_fullsized":"2023-12-13T23:55:14.624340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613654+0000","last_change":"2023-12-13T23:55:11.765823+0000","last_active":"2023-12-13T23:55:14.613654+0000","last_peered":"2023-12-13T23:55:14.613654+0000","last_clean":"2023-12-13T23:55:14.613654+0000","last_became_active":"2023-12-13T23:55:11.765515+0000","last_became_peered":"2023-12-13T23:55:11.765515+0000","last_unstale":"2023-12-13T23:55:14.613654+0000","last_undegraded":"2023-12-13T23:55:14.613654+0000","last_fullsized":"2023-12-13T23:55:14.613654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741390+0000","last_change":"2023-12-13T23:55:12.750387+0000","last_active":"2023-12-13T23:55:13.741390+0000","last_peered":"2023-12-13T23:55:13.741390+0000","last_clean":"2023-12-13T23:55:13.741390+0000","last_became_active":"2023-12-13T23:55:12.749101+0000","last_became_peered":"2023-12-13T23:55:12.749101+0000","last_unstale":"2023-12-13T23:55:13.741390+0000","last_undegraded":"2023-12-13T23:55:13.741390+0000","last_fullsized":"2023-12-13T23:55:13.741390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624137+0000","last_change":"2023-12-13T23:55:11.763071+0000","last_active":"2023-12-13T23:55:14.624137+0000","last_peered":"2023-12-13T23:55:14.624137+0000","last_clean":"2023-12-13T23:55:14.624137+0000","last_became_active":"2023-12-13T23:55:11.762774+0000","last_became_peered":"2023-12-13T23:55:11.762774+0000","last_unstale":"2023-12-13T23:55:14.624137+0000","last_undegraded":"2023-12-13T23:55:14.624137+0000","last_fullsized":"2023-12-13T23:55:14.624137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740649+0000","last_change":"2023-12-13T23:55:12.747891+0000","last_active":"2023-12-13T23:55:13.740649+0000","last_peered":"2023-12-13T23:55:13.740649+0000","last_clean":"2023-12-13T23:55:13.740649+0000","last_became_active":"2023-12-13T23:55:12.747629+0000","last_became_peered":"2023-12-13T23:55:12.747629+0000","last_unstale":"2023-12-13T23:55:13.740649+0000","last_undegraded":"2023-12-13T23:55:13.740649+0000","last_fullsized":"2023-12-13T23:55:13.740649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624263+0000","last_change":"2023-12-13T23:55:11.768277+0000","last_active":"2023-12-13T23:55:14.624263+0000","last_peered":"2023-12-13T23:55:14.624263+0000","last_clean":"2023-12-13T23:55:14.624263+0000","last_became_active":"2023-12-13T23:55:11.768039+0000","last_became_peered":"2023-12-13T23:55:11.768039+0000","last_unstale":"2023-12-13T23:55:14.624263+0000","last_undegraded":"2023-12-13T23:55:14.624263+0000","last_fullsized":"2023-12-13T23:55:14.624263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624351+0000","last_change":"2023-12-13T23:55:12.758134+0000","last_active":"2023-12-13T23:55:14.624351+0000","last_peered":"2023-12-13T23:55:14.624351+0000","last_clean":"2023-12-13T23:55:14.624351+0000","last_became_active":"2023-12-13T23:55:12.757679+0000","last_became_peered":"2023-12-13T23:55:12.757679+0000","last_unstale":"2023-12-13T23:55:14.624351+0000","last_undegraded":"2023-12-13T23:55:14.624351+0000","last_fullsized":"2023-12-13T23:55:14.624351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625036+0000","last_change":"2023-12-13T23:55:11.765748+0000","last_active":"2023-12-13T23:55:14.625036+0000","last_peered":"2023-12-13T23:55:14.625036+0000","last_clean":"2023-12-13T23:55:14.625036+0000","last_became_active":"2023-12-13T23:55:11.765320+0000","last_became_peered":"2023-12-13T23:55:11.765320+0000","last_unstale":"2023-12-13T23:55:14.625036+0000","last_undegraded":"2023-12-13T23:55:14.625036+0000","last_fullsized":"2023-12-13T23:55:14.625036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740530+0000","last_change":"2023-12-13T23:55:12.747127+0000","last_active":"2023-12-13T23:55:13.740530+0000","last_peered":"2023-12-13T23:55:13.740530+0000","last_clean":"2023-12-13T23:55:13.740530+0000","last_became_active":"2023-12-13T23:55:12.746838+0000","last_became_peered":"2023-12-13T23:55:12.746838+0000","last_unstale":"2023-12-13T23:55:13.740530+0000","last_undegraded":"2023-12-13T23:55:13.740530+0000","last_fullsized":"2023-12-13T23:55:13.740530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623961+0000","last_change":"2023-12-13T23:55:11.765098+0000","last_active":"2023-12-13T23:55:14.623961+0000","last_peered":"2023-12-13T23:55:14.623961+0000","last_clean":"2023-12-13T23:55:14.623961+0000","last_became_active":"2023-12-13T23:55:11.764746+0000","last_became_peered":"2023-12-13T23:55:11.764746+0000","last_unstale":"2023-12-13T23:55:14.623961+0000","last_undegraded":"2023-12-13T23:55:14.623961+0000","last_fullsized":"2023-12-13T23:55:14.623961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624195+0000","last_change":"2023-12-13T23:55:12.759703+0000","last_active":"2023-12-13T23:55:14.624195+0000","last_peered":"2023-12-13T23:55:14.624195+0000","last_clean":"2023-12-13T23:55:14.624195+0000","last_became_active":"2023-12-13T23:55:12.759371+0000","last_became_peered":"2023-12-13T23:55:12.759371+0000","last_unstale":"2023-12-13T23:55:14.624195+0000","last_undegraded":"2023-12-13T23:55:14.624195+0000","last_fullsized":"2023-12-13T23:55:14.624195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623693+0000","last_change":"2023-12-13T23:55:11.757065+0000","last_active":"2023-12-13T23:55:14.623693+0000","last_peered":"2023-12-13T23:55:14.623693+0000","last_clean":"2023-12-13T23:55:14.623693+0000","last_became_active":"2023-12-13T23:55:11.756732+0000","last_became_peered":"2023-12-13T23:55:11.756732+0000","last_unstale":"2023-12-13T23:55:14.623693+0000","last_undegraded":"2023-12-13T23:55:14.623693+0000","last_fullsized":"2023-12-13T23:55:14.623693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740938+0000","last_change":"2023-12-13T23:55:12.747168+0000","last_active":"2023-12-13T23:55:13.740938+0000","last_peered":"2023-12-13T23:55:13.740938+0000","last_clean":"2023-12-13T23:55:13.740938+0000","last_became_active":"2023-12-13T23:55:12.746860+0000","last_became_peered":"2023-12-13T23:55:12.746860+0000","last_unstale":"2023-12-13T23:55:13.740938+0000","last_undegraded":"2023-12-13T23:55:13.740938+0000","last_fullsized":"2023-12-13T23:55:13.740938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740851+0000","last_change":"2023-12-13T23:55:11.765922+0000","last_active":"2023-12-13T23:55:13.740851+0000","last_peered":"2023-12-13T23:55:13.740851+0000","last_clean":"2023-12-13T23:55:13.740851+0000","last_became_active":"2023-12-13T23:55:11.765641+0000","last_became_peered":"2023-12-13T23:55:11.765641+0000","last_unstale":"2023-12-13T23:55:13.740851+0000","last_undegraded":"2023-12-13T23:55:13.740851+0000","last_fullsized":"2023-12-13T23:55:13.740851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:07:21.569553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741605+0000","last_change":"2023-12-13T23:55:12.747433+0000","last_active":"2023-12-13T23:55:13.741605+0000","last_peered":"2023-12-13T23:55:13.741605+0000","last_clean":"2023-12-13T23:55:13.741605+0000","last_became_active":"2023-12-13T23:55:12.747169+0000","last_became_peered":"2023-12-13T23:55:12.747169+0000","last_unstale":"2023-12-13T23:55:13.741605+0000","last_undegraded":"2023-12-13T23:55:13.741605+0000","last_fullsized":"2023-12-13T23:55:13.741605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:03:57.776819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624072+0000","last_change":"2023-12-13T23:55:11.762985+0000","last_active":"2023-12-13T23:55:14.624072+0000","last_peered":"2023-12-13T23:55:14.624072+0000","last_clean":"2023-12-13T23:55:14.624072+0000","last_became_active":"2023-12-13T23:55:11.762654+0000","last_became_peered":"2023-12-13T23:55:11.762654+0000","last_unstale":"2023-12-13T23:55:14.624072+0000","last_undegraded":"2023-12-13T23:55:14.624072+0000","last_fullsized":"2023-12-13T23:55:14.624072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741322+0000","last_change":"2023-12-13T23:55:12.756265+0000","last_active":"2023-12-13T23:55:13.741322+0000","last_peered":"2023-12-13T23:55:13.741322+0000","last_clean":"2023-12-13T23:55:13.741322+0000","last_became_active":"2023-12-13T23:55:12.756003+0000","last_became_peered":"2023-12-13T23:55:12.756003+0000","last_unstale":"2023-12-13T23:55:13.741322+0000","last_undegraded":"2023-12-13T23:55:13.741322+0000","last_fullsized":"2023-12-13T23:55:13.741322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:26:45.885932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613709+0000","last_change":"2023-12-13T23:55:11.766945+0000","last_active":"2023-12-13T23:55:14.613709+0000","last_peered":"2023-12-13T23:55:14.613709+0000","last_clean":"2023-12-13T23:55:14.613709+0000","last_became_active":"2023-12-13T23:55:11.766671+0000","last_became_peered":"2023-12-13T23:55:11.766671+0000","last_unstale":"2023-12-13T23:55:14.613709+0000","last_undegraded":"2023-12-13T23:55:14.613709+0000","last_fullsized":"2023-12-13T23:55:14.613709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624646+0000","last_change":"2023-12-13T23:55:12.752034+0000","last_active":"2023-12-13T23:55:14.624646+0000","last_peered":"2023-12-13T23:55:14.624646+0000","last_clean":"2023-12-13T23:55:14.624646+0000","last_became_active":"2023-12-13T23:55:12.751764+0000","last_became_peered":"2023-12-13T23:55:12.751764+0000","last_unstale":"2023-12-13T23:55:14.624646+0000","last_undegraded":"2023-12-13T23:55:14.624646+0000","last_fullsized":"2023-12-13T23:55:14.624646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623912+0000","last_change":"2023-12-13T23:55:11.765879+0000","last_active":"2023-12-13T23:55:14.623912+0000","last_peered":"2023-12-13T23:55:14.623912+0000","last_clean":"2023-12-13T23:55:14.623912+0000","last_became_active":"2023-12-13T23:55:11.765607+0000","last_became_peered":"2023-12-13T23:55:11.765607+0000","last_unstale":"2023-12-13T23:55:14.623912+0000","last_undegraded":"2023-12-13T23:55:14.623912+0000","last_fullsized":"2023-12-13T23:55:14.623912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:40.262294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740784+0000","last_change":"2023-12-13T23:55:12.756349+0000","last_active":"2023-12-13T23:55:13.740784+0000","last_peered":"2023-12-13T23:55:13.740784+0000","last_clean":"2023-12-13T23:55:13.740784+0000","last_became_active":"2023-12-13T23:55:12.756033+0000","last_became_peered":"2023-12-13T23:55:12.756033+0000","last_unstale":"2023-12-13T23:55:13.740784+0000","last_undegraded":"2023-12-13T23:55:13.740784+0000","last_fullsized":"2023-12-13T23:55:13.740784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-13T23:55:12.739242+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:12.739242+0000","last_peered":"2023-12-13T23:55:12.739242+0000","last_clean":"2023-12-13T23:55:12.739242+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:12.739242+0000","last_undegraded":"2023-12-13T23:55:12.739242+0000","last_fullsized":"2023-12-13T23:55:12.739242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624508+0000","last_change":"2023-12-13T23:55:11.758971+0000","last_active":"2023-12-13T23:55:14.624508+0000","last_peered":"2023-12-13T23:55:14.624508+0000","last_clean":"2023-12-13T23:55:14.624508+0000","last_became_active":"2023-12-13T23:55:11.758594+0000","last_became_peered":"2023-12-13T23:55:11.758594+0000","last_unstale":"2023-12-13T23:55:14.624508+0000","last_undegraded":"2023-12-13T23:55:14.624508+0000","last_fullsized":"2023-12-13T23:55:14.624508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737699+0000","last_change":"2023-12-13T23:55:12.755959+0000","last_active":"2023-12-13T23:55:13.737699+0000","last_peered":"2023-12-13T23:55:13.737699+0000","last_clean":"2023-12-13T23:55:13.737699+0000","last_became_active":"2023-12-13T23:55:12.755662+0000","last_became_peered":"2023-12-13T23:55:12.755662+0000","last_unstale":"2023-12-13T23:55:13.737699+0000","last_undegraded":"2023-12-13T23:55:13.737699+0000","last_fullsized":"2023-12-13T23:55:13.737699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740316+0000","last_change":"2023-12-13T23:55:11.767937+0000","last_active":"2023-12-13T23:55:13.740316+0000","last_peered":"2023-12-13T23:55:13.740316+0000","last_clean":"2023-12-13T23:55:13.740316+0000","last_became_active":"2023-12-13T23:55:11.767613+0000","last_became_peered":"2023-12-13T23:55:11.767613+0000","last_unstale":"2023-12-13T23:55:13.740316+0000","last_undegraded":"2023-12-13T23:55:13.740316+0000","last_fullsized":"2023-12-13T23:55:13.740316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:53.188642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624112+0000","last_change":"2023-12-13T23:55:12.757755+0000","last_active":"2023-12-13T23:55:14.624112+0000","last_peered":"2023-12-13T23:55:14.624112+0000","last_clean":"2023-12-13T23:55:14.624112+0000","last_became_active":"2023-12-13T23:55:12.757456+0000","last_became_peered":"2023-12-13T23:55:12.757456+0000","last_unstale":"2023-12-13T23:55:14.624112+0000","last_undegraded":"2023-12-13T23:55:14.624112+0000","last_fullsized":"2023-12-13T23:55:14.624112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623940+0000","last_change":"2023-12-13T23:55:11.755829+0000","last_active":"2023-12-13T23:55:14.623940+0000","last_peered":"2023-12-13T23:55:14.623940+0000","last_clean":"2023-12-13T23:55:14.623940+0000","last_became_active":"2023-12-13T23:55:11.755573+0000","last_became_peered":"2023-12-13T23:55:11.755573+0000","last_unstale":"2023-12-13T23:55:14.623940+0000","last_undegraded":"2023-12-13T23:55:14.623940+0000","last_fullsized":"2023-12-13T23:55:14.623940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:56.833424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613504+0000","last_change":"2023-12-13T23:55:12.749732+0000","last_active":"2023-12-13T23:55:14.613504+0000","last_peered":"2023-12-13T23:55:14.613504+0000","last_clean":"2023-12-13T23:55:14.613504+0000","last_became_active":"2023-12-13T23:55:12.749473+0000","last_became_peered":"2023-12-13T23:55:12.749473+0000","last_unstale":"2023-12-13T23:55:14.613504+0000","last_undegraded":"2023-12-13T23:55:14.613504+0000","last_fullsized":"2023-12-13T23:55:14.613504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625041+0000","last_change":"2023-12-13T23:55:11.765970+0000","last_active":"2023-12-13T23:55:14.625041+0000","last_peered":"2023-12-13T23:55:14.625041+0000","last_clean":"2023-12-13T23:55:14.625041+0000","last_became_active":"2023-12-13T23:55:11.765629+0000","last_became_peered":"2023-12-13T23:55:11.765629+0000","last_unstale":"2023-12-13T23:55:14.625041+0000","last_undegraded":"2023-12-13T23:55:14.625041+0000","last_fullsized":"2023-12-13T23:55:14.625041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741379+0000","last_change":"2023-12-13T23:55:12.758873+0000","last_active":"2023-12-13T23:55:13.741379+0000","last_peered":"2023-12-13T23:55:13.741379+0000","last_clean":"2023-12-13T23:55:13.741379+0000","last_became_active":"2023-12-13T23:55:12.758618+0000","last_became_peered":"2023-12-13T23:55:12.758618+0000","last_unstale":"2023-12-13T23:55:13.741379+0000","last_undegraded":"2023-12-13T23:55:13.741379+0000","last_fullsized":"2023-12-13T23:55:13.741379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613597+0000","last_change":"2023-12-13T23:55:11.762944+0000","last_active":"2023-12-13T23:55:14.613597+0000","last_peered":"2023-12-13T23:55:14.613597+0000","last_clean":"2023-12-13T23:55:14.613597+0000","last_became_active":"2023-12-13T23:55:11.762691+0000","last_became_peered":"2023-12-13T23:55:11.762691+0000","last_unstale":"2023-12-13T23:55:14.613597+0000","last_undegraded":"2023-12-13T23:55:14.613597+0000","last_fullsized":"2023-12-13T23:55:14.613597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625017+0000","last_change":"2023-12-13T23:55:12.755585+0000","last_active":"2023-12-13T23:55:14.625017+0000","last_peered":"2023-12-13T23:55:14.625017+0000","last_clean":"2023-12-13T23:55:14.625017+0000","last_became_active":"2023-12-13T23:55:12.755284+0000","last_became_peered":"2023-12-13T23:55:12.755284+0000","last_unstale":"2023-12-13T23:55:14.625017+0000","last_undegraded":"2023-12-13T23:55:14.625017+0000","last_fullsized":"2023-12-13T23:55:14.625017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741015+0000","last_change":"2023-12-13T23:55:11.770291+0000","last_active":"2023-12-13T23:55:13.741015+0000","last_peered":"2023-12-13T23:55:13.741015+0000","last_clean":"2023-12-13T23:55:13.741015+0000","last_became_active":"2023-12-13T23:55:11.769672+0000","last_became_peered":"2023-12-13T23:55:11.769672+0000","last_unstale":"2023-12-13T23:55:13.741015+0000","last_undegraded":"2023-12-13T23:55:13.741015+0000","last_fullsized":"2023-12-13T23:55:13.741015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613615+0000","last_change":"2023-12-13T23:55:12.749467+0000","last_active":"2023-12-13T23:55:14.613615+0000","last_peered":"2023-12-13T23:55:14.613615+0000","last_clean":"2023-12-13T23:55:14.613615+0000","last_became_active":"2023-12-13T23:55:12.749185+0000","last_became_peered":"2023-12-13T23:55:12.749185+0000","last_unstale":"2023-12-13T23:55:14.613615+0000","last_undegraded":"2023-12-13T23:55:14.613615+0000","last_fullsized":"2023-12-13T23:55:14.613615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613740+0000","last_change":"2023-12-13T23:55:11.763643+0000","last_active":"2023-12-13T23:55:14.613740+0000","last_peered":"2023-12-13T23:55:14.613740+0000","last_clean":"2023-12-13T23:55:14.613740+0000","last_became_active":"2023-12-13T23:55:11.763378+0000","last_became_peered":"2023-12-13T23:55:11.763378+0000","last_unstale":"2023-12-13T23:55:14.613740+0000","last_undegraded":"2023-12-13T23:55:14.613740+0000","last_fullsized":"2023-12-13T23:55:14.613740+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737984+0000","last_change":"2023-12-13T23:55:12.752448+0000","last_active":"2023-12-13T23:55:13.737984+0000","last_peered":"2023-12-13T23:55:13.737984+0000","last_clean":"2023-12-13T23:55:13.737984+0000","last_became_active":"2023-12-13T23:55:12.752155+0000","last_became_peered":"2023-12-13T23:55:12.752155+0000","last_unstale":"2023-12-13T23:55:13.737984+0000","last_undegraded":"2023-12-13T23:55:13.737984+0000","last_fullsized":"2023-12-13T23:55:13.737984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624790+0000","last_change":"2023-12-13T23:55:11.768156+0000","last_active":"2023-12-13T23:55:14.624790+0000","last_peered":"2023-12-13T23:55:14.624790+0000","last_clean":"2023-12-13T23:55:14.624790+0000","last_became_active":"2023-12-13T23:55:11.767878+0000","last_became_peered":"2023-12-13T23:55:11.767878+0000","last_unstale":"2023-12-13T23:55:14.624790+0000","last_undegraded":"2023-12-13T23:55:14.624790+0000","last_fullsized":"2023-12-13T23:55:14.624790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624226+0000","last_change":"2023-12-13T23:55:12.751906+0000","last_active":"2023-12-13T23:55:14.624226+0000","last_peered":"2023-12-13T23:55:14.624226+0000","last_clean":"2023-12-13T23:55:14.624226+0000","last_became_active":"2023-12-13T23:55:12.751468+0000","last_became_peered":"2023-12-13T23:55:12.751468+0000","last_unstale":"2023-12-13T23:55:14.624226+0000","last_undegraded":"2023-12-13T23:55:14.624226+0000","last_fullsized":"2023-12-13T23:55:14.624226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740225+0000","last_change":"2023-12-13T23:55:11.768417+0000","last_active":"2023-12-13T23:55:13.740225+0000","last_peered":"2023-12-13T23:55:13.740225+0000","last_clean":"2023-12-13T23:55:13.740225+0000","last_became_active":"2023-12-13T23:55:11.768141+0000","last_became_peered":"2023-12-13T23:55:11.768141+0000","last_unstale":"2023-12-13T23:55:13.740225+0000","last_undegraded":"2023-12-13T23:55:13.740225+0000","last_fullsized":"2023-12-13T23:55:13.740225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740516+0000","last_change":"2023-12-13T23:55:12.751650+0000","last_active":"2023-12-13T23:55:13.740516+0000","last_peered":"2023-12-13T23:55:13.740516+0000","last_clean":"2023-12-13T23:55:13.740516+0000","last_became_active":"2023-12-13T23:55:12.749884+0000","last_became_peered":"2023-12-13T23:55:12.749884+0000","last_unstale":"2023-12-13T23:55:13.740516+0000","last_undegraded":"2023-12-13T23:55:13.740516+0000","last_fullsized":"2023-12-13T23:55:13.740516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:12.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613607+0000","last_change":"2023-12-13T23:55:11.761362+0000","last_active":"2023-12-13T23:55:14.613607+0000","last_peered":"2023-12-13T23:55:14.613607+0000","last_clean":"2023-12-13T23:55:14.613607+0000","last_became_active":"2023-12-13T23:55:11.761091+0000","last_became_peered":"2023-12-13T23:55:11.761091+0000","last_unstale":"2023-12-13T23:55:14.613607+0000","last_undegraded":"2023-12-13T23:55:14.613607+0000","last_fullsized":"2023-12-13T23:55:14.613607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:45.292370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737621+0000","last_change":"2023-12-13T23:55:12.747019+0000","last_active":"2023-12-13T23:55:13.737621+0000","last_peered":"2023-12-13T23:55:13.737621+0000","last_clean":"2023-12-13T23:55:13.737621+0000","last_became_active":"2023-12-13T23:55:12.746720+0000","last_became_peered":"2023-12-13T23:55:12.746720+0000","last_unstale":"2023-12-13T23:55:13.737621+0000","last_undegraded":"2023-12-13T23:55:13.737621+0000","last_fullsized":"2023-12-13T23:55:13.737621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613672+0000","last_change":"2023-12-13T23:55:11.761235+0000","last_active":"2023-12-13T23:55:14.613672+0000","last_peered":"2023-12-13T23:55:14.613672+0000","last_clean":"2023-12-13T23:55:14.613672+0000","last_became_active":"2023-12-13T23:55:11.760961+0000","last_became_peered":"2023-12-13T23:55:11.760961+0000","last_unstale":"2023-12-13T23:55:14.613672+0000","last_undegraded":"2023-12-13T23:55:14.613672+0000","last_fullsized":"2023-12-13T23:55:14.613672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737873+0000","last_change":"2023-12-13T23:55:12.757945+0000","last_active":"2023-12-13T23:55:13.737873+0000","last_peered":"2023-12-13T23:55:13.737873+0000","last_clean":"2023-12-13T23:55:13.737873+0000","last_became_active":"2023-12-13T23:55:12.757685+0000","last_became_peered":"2023-12-13T23:55:12.757685+0000","last_unstale":"2023-12-13T23:55:13.737873+0000","last_undegraded":"2023-12-13T23:55:13.737873+0000","last_fullsized":"2023-12-13T23:55:13.737873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740608+0000","last_change":"2023-12-13T23:55:11.769857+0000","last_active":"2023-12-13T23:55:13.740608+0000","last_peered":"2023-12-13T23:55:13.740608+0000","last_clean":"2023-12-13T23:55:13.740608+0000","last_became_active":"2023-12-13T23:55:11.769503+0000","last_became_peered":"2023-12-13T23:55:11.769503+0000","last_unstale":"2023-12-13T23:55:13.740608+0000","last_undegraded":"2023-12-13T23:55:13.740608+0000","last_fullsized":"2023-12-13T23:55:13.740608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624730+0000","last_change":"2023-12-13T23:55:12.747991+0000","last_active":"2023-12-13T23:55:14.624730+0000","last_peered":"2023-12-13T23:55:14.624730+0000","last_clean":"2023-12-13T23:55:14.624730+0000","last_became_active":"2023-12-13T23:55:12.746787+0000","last_became_peered":"2023-12-13T23:55:12.746787+0000","last_unstale":"2023-12-13T23:55:14.624730+0000","last_undegraded":"2023-12-13T23:55:14.624730+0000","last_fullsized":"2023-12-13T23:55:14.624730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624966+0000","last_change":"2023-12-13T23:55:12.755883+0000","last_active":"2023-12-13T23:55:14.624966+0000","last_peered":"2023-12-13T23:55:14.624966+0000","last_clean":"2023-12-13T23:55:14.624966+0000","last_became_active":"2023-12-13T23:55:12.755573+0000","last_became_peered":"2023-12-13T23:55:12.755573+0000","last_unstale":"2023-12-13T23:55:14.624966+0000","last_undegraded":"2023-12-13T23:55:14.624966+0000","last_fullsized":"2023-12-13T23:55:14.624966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741033+0000","last_change":"2023-12-13T23:55:11.769532+0000","last_active":"2023-12-13T23:55:13.741033+0000","last_peered":"2023-12-13T23:55:13.741033+0000","last_clean":"2023-12-13T23:55:13.741033+0000","last_became_active":"2023-12-13T23:55:11.769251+0000","last_became_peered":"2023-12-13T23:55:11.769251+0000","last_unstale":"2023-12-13T23:55:13.741033+0000","last_undegraded":"2023-12-13T23:55:13.741033+0000","last_fullsized":"2023-12-13T23:55:13.741033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624446+0000","last_change":"2023-12-13T23:55:12.757891+0000","last_active":"2023-12-13T23:55:14.624446+0000","last_peered":"2023-12-13T23:55:14.624446+0000","last_clean":"2023-12-13T23:55:14.624446+0000","last_became_active":"2023-12-13T23:55:12.757643+0000","last_became_peered":"2023-12-13T23:55:12.757643+0000","last_unstale":"2023-12-13T23:55:14.624446+0000","last_undegraded":"2023-12-13T23:55:14.624446+0000","last_fullsized":"2023-12-13T23:55:14.624446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740911+0000","last_change":"2023-12-13T23:55:11.769811+0000","last_active":"2023-12-13T23:55:13.740911+0000","last_peered":"2023-12-13T23:55:13.740911+0000","last_clean":"2023-12-13T23:55:13.740911+0000","last_became_active":"2023-12-13T23:55:11.769483+0000","last_became_peered":"2023-12-13T23:55:11.769483+0000","last_unstale":"2023-12-13T23:55:13.740911+0000","last_undegraded":"2023-12-13T23:55:13.740911+0000","last_fullsized":"2023-12-13T23:55:13.740911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624275+0000","last_change":"2023-12-13T23:55:12.758511+0000","last_active":"2023-12-13T23:55:14.624275+0000","last_peered":"2023-12-13T23:55:14.624275+0000","last_clean":"2023-12-13T23:55:14.624275+0000","last_became_active":"2023-12-13T23:55:12.757824+0000","last_became_peered":"2023-12-13T23:55:12.757824+0000","last_unstale":"2023-12-13T23:55:14.624275+0000","last_undegraded":"2023-12-13T23:55:14.624275+0000","last_fullsized":"2023-12-13T23:55:14.624275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737890+0000","last_change":"2023-12-13T23:55:11.761808+0000","last_active":"2023-12-13T23:55:13.737890+0000","last_peered":"2023-12-13T23:55:13.737890+0000","last_clean":"2023-12-13T23:55:13.737890+0000","last_became_active":"2023-12-13T23:55:11.761559+0000","last_became_peered":"2023-12-13T23:55:11.761559+0000","last_unstale":"2023-12-13T23:55:13.737890+0000","last_undegraded":"2023-12-13T23:55:13.737890+0000","last_fullsized":"2023-12-13T23:55:13.737890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737814+0000","last_change":"2023-12-13T23:55:12.748462+0000","last_active":"2023-12-13T23:55:13.737814+0000","last_peered":"2023-12-13T23:55:13.737814+0000","last_clean":"2023-12-13T23:55:13.737814+0000","last_became_active":"2023-12-13T23:55:12.748166+0000","last_became_peered":"2023-12-13T23:55:12.748166+0000","last_unstale":"2023-12-13T23:55:13.737814+0000","last_undegraded":"2023-12-13T23:55:13.737814+0000","last_fullsized":"2023-12-13T23:55:13.737814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624331+0000","last_change":"2023-12-13T23:55:11.771111+0000","last_active":"2023-12-13T23:55:14.624331+0000","last_peered":"2023-12-13T23:55:14.624331+0000","last_clean":"2023-12-13T23:55:14.624331+0000","last_became_active":"2023-12-13T23:55:11.770795+0000","last_became_peered":"2023-12-13T23:55:11.770795+0000","last_unstale":"2023-12-13T23:55:14.624331+0000","last_undegraded":"2023-12-13T23:55:14.624331+0000","last_fullsized":"2023-12-13T23:55:14.624331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":70893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":69371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":69281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.245}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.22600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":212992,"data_stored":69441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":811008,"data_stored":660287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:20.382 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-13T23:55:20.382 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-13T23:55:20.383 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-13T23:55:20.383 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-13T23:55:20.383 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-13T23:55:20.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.519+0000 7ff69eb76640 1 Processor -- start 2023-12-13T23:55:20.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.519+0000 7ff69eb76640 1 -- start start 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69eb76640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 0x7ff6980b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69eb76640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff6980d2910 0x7ff6980d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69eb76640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6980b2b10 con 0x7ff6980d3730 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69eb76640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6980d3bc0 con 0x7ff6980d2910 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69eb76640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6980d1590 con 0x7ff69815d330 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff697fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff6980d2910 0x7ff6980d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff697fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff6980d2910 0x7ff6980d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57002/0 (socket says 172.21.15.124:57002) 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff697fff640 1 -- 172.21.15.124:0/1753892400 learned_addr learned my addr 172.21.15.124:0/1753892400 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.520+0000 7ff69c912640 1 --2- 172.21.15.124:0/1753892400 >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 0x7ff6980b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff697fff640 1 -- 172.21.15.124:0/1753892400 >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 msgr2=0x7ff6980b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff697fff640 1 --2- 172.21.15.124:0/1753892400 >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 0x7ff6980b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff697fff640 1 -- 172.21.15.124:0/1753892400 >> v1:172.21.15.203:6789/0 conn(0x7ff6980d3730 legacy=0x7ff6980b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff697fff640 1 -- 172.21.15.124:0/1753892400 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6980d1870 con 0x7ff6980d2910 2023-12-13T23:55:20.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff69c912640 1 --2- 172.21.15.124:0/1753892400 >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 0x7ff6980b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:20.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff697fff640 1 --2- 172.21.15.124:0/1753892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff6980d31f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6980d0d10 tx=0x7ff68c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fd54533f638d5e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.521+0000 7ff6977fe640 1 -- 172.21.15.124:0/1753892400 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c031870 con 0x7ff6980d2910 2023-12-13T23:55:20.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff6977fe640 1 -- 172.21.15.124:0/1753892400 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c0319d0 con 0x7ff6980d2910 2023-12-13T23:55:20.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff6977fe640 1 -- 172.21.15.124:0/1753892400 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c031d60 con 0x7ff6980d2910 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 -- 172.21.15.124:0/1753892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 msgr2=0x7ff6980d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 --2- 172.21.15.124:0/1753892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff6980d31f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff6980d0d10 tx=0x7ff68c02b910 comp rx=0 tx=0).stop 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 -- 172.21.15.124:0/1753892400 shutdown_connections 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 --2- 172.21.15.124:0/1753892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff6980d31f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 --2- 172.21.15.124:0/1753892400 >> v2:172.21.15.203:3300/0 conn(0x7ff69815d330 0x7ff6980b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 -- 172.21.15.124:0/1753892400 >> 172.21.15.124:0/1753892400 conn(0x7ff698072020 msgr2=0x7ff698072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 -- 172.21.15.124:0/1753892400 shutdown_connections 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.522+0000 7ff69eb76640 1 -- 172.21.15.124:0/1753892400 wait complete. 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.523+0000 7ff69eb76640 1 Processor -- start 2023-12-13T23:55:20.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.523+0000 7ff69eb76640 1 -- start start 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 0x7ff6980c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff69815d330 0x7ff69814b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980c73d0 con 0x7ff6980d2910 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980c76b0 con 0x7ff6980d3730 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69eb76640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6980c7990 con 0x7ff69815d330 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69c912640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69c912640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57004/0 (socket says 172.21.15.124:57004) 2023-12-13T23:55:20.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69c912640 1 -- 172.21.15.124:0/947828152 learned_addr learned my addr 172.21.15.124:0/947828152 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff697fff640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 0x7ff6980c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.524+0000 7ff69d113640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff69815d330 0x7ff69814b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 -- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff69815d330 msgr2=0x7ff69814b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff69815d330 0x7ff69814b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 -- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 msgr2=0x7ff6980c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 0x7ff6980c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 -- 172.21.15.124:0/947828152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c0094d0 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff697fff640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 0x7ff6980c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69c912640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff68800d7d0 tx=0x7ff68800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688004030 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff688004190 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6980c7c70 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688004520 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.525+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6980c70e0 con 0x7ff6980d2910 2023-12-13T23:55:20.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.526+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff698069400 con 0x7ff6980d2910 2023-12-13T23:55:20.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.526+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff6880096a0 con 0x7ff6980d2910 2023-12-13T23:55:20.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.527+0000 7ff6957fa640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 0x7ff68406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.527+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff688089c10 con 0x7ff6980d2910 2023-12-13T23:55:20.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.527+0000 7ff697fff640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 0x7ff68406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.527+0000 7ff697fff640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 0x7ff68406d680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff68c036000 tx=0x7ff68c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.531+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff688054940 con 0x7ff6980d2910 2023-12-13T23:55:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.742+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff6980d3290 con 0x7ff6980d2910 2023-12-13T23:55:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.745+0000 7ff6957fa640 1 -- 172.21.15.124:0/947828152 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171170 (secure 0 0 0) 0x7ff688057f60 con 0x7ff6980d2910 2023-12-13T23:55:20.745 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:20.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.747+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 msgr2=0x7ff68406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.747+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 0x7ff68406d680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff68c036000 tx=0x7ff68c03c040 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.747+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 msgr2=0x7ff698154c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.747+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff68800d7d0 tx=0x7ff68800dc90 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 shutdown_connections 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff68406b1c0 0x7ff68406d680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff6980d3730 0x7ff6980c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff69815d330 0x7ff69814b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 --2- 172.21.15.124:0/947828152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6980d2910 0x7ff698154c90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 >> 172.21.15.124:0/947828152 conn(0x7ff698072020 msgr2=0x7ff6980d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 shutdown_connections 2023-12-13T23:55:20.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.748+0000 7ff69eb76640 1 -- 172.21.15.124:0/947828152 wait complete. 2023-12-13T23:55:20.757 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":5,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6832","nonce":4285908483},{"type":"v1","addr":"172.21.15.203:6833","nonce":4285908483}]},"active_addr":"172.21.15.203:6833/4285908483","active_change":"2023-12-13T23:54:52.098598+0000","active_mgr_features":4540701272860131327,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540701272860131327,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1227005828}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":1715253684}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3610229876}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.203:0","nonce":3984545708}]}]} 2023-12-13T23:55:20.761 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-13T23:55:20.761 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-13T23:55:20.761 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:20.860 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.862+0000 7f8da374c640 1 Processor -- start 2023-12-13T23:55:20.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 -- start start 2023-12-13T23:55:20.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0d2910 0x7f8d9c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d9c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c0b2b10 con 0x7f8d9c15d330 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9c0d1590 con 0x7f8d9c0d2910 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da374c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0d1870 con 0x7f8d9c0d3730 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da14e8640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8d9c15d330 0x7f8d9c0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43548/0 (socket says 172.21.15.124:43548) 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.863+0000 7f8da14e8640 1 -- 172.21.15.124:0/176360041 learned_addr learned my addr 172.21.15.124:0/176360041 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160282756 0 0) 0x7f8d9c0b2b10 con 0x7f8d9c15d330 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d88003520 con 0x7f8d9c15d330 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d9c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da0ce7640 1 --2- 172.21.15.124:0/176360041 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0d2910 0x7f8d9c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682103465 0 0) 0x7f8d88003520 con 0x7f8d9c15d330 2023-12-13T23:55:20.865 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 msgr2=0x7f8d9c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d9c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0d2910 msgr2=0x7f8d9c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 --2- 172.21.15.124:0/176360041 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0d2910 0x7f8d9c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c0d3b90 con 0x7f8d9c15d330 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8d980019f0 con 0x7f8d9c15d330 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 >> v1:172.21.15.124:6789/0 conn(0x7f8d9c15d330 legacy=0x7f8d9c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d88003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88003ef0 0x7f8d880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d88004d00 0x7f8d88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0d1590 con 0x7f8d88004d00 2023-12-13T23:55:20.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0d1870 con 0x7f8d88003ef0 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8800b9d0 con 0x7f8d9c0d3730 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da0ce7640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88003ef0 0x7f8d880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d88004d00 0x7f8d88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da14e8640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d88003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 -- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 msgr2=0x7f8d88003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0d3730 0x7f8d88003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 -- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88003ef0 msgr2=0x7f8d880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d88003ef0 0x7f8d880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.864+0000 7f8da1ce9640 1 -- 172.21.15.124:0/176360041 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d88003520 con 0x7f8d88004d00 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.865+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d88004d00 0x7f8d88005180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009af0 tx=0x7f8d880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f68baf9dbf426cb4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.865+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9402c6b0 con 0x7f8d88004d00 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.865+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9402c810 con 0x7f8d88004d00 2023-12-13T23:55:20.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.865+0000 7f8d93fff640 1 -- 172.21.15.124:0/176360041 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9402cba0 con 0x7f8d88004d00 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 -- 172.21.15.124:0/176360041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d88004d00 msgr2=0x7f8d88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 --2- 172.21.15.124:0/176360041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d88004d00 0x7f8d88005180 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f8d94009af0 tx=0x7f8d880057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da1ce9640 1 -- 172.21.15.124:0/176360041 reap_dead start 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 -- 172.21.15.124:0/176360041 shutdown_connections 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 -- 172.21.15.124:0/176360041 >> 172.21.15.124:0/176360041 conn(0x7f8d9c072020 msgr2=0x7f8d9c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 -- 172.21.15.124:0/176360041 shutdown_connections 2023-12-13T23:55:20.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.866+0000 7f8da374c640 1 -- 172.21.15.124:0/176360041 wait complete. 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.867+0000 7f8da374c640 1 Processor -- start 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.867+0000 7f8da374c640 1 -- start start 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0c5d00 0x7f8d9c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0c60e0 0x7f8d9c157410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c158700 con 0x7f8d9c14ae90 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c0cbdb0 con 0x7f8d9c0c60e0 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c158140 con 0x7f8d9c0c5d00 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57018/0 (socket says 172.21.15.124:57018) 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 -- 172.21.15.124:0/1817907082 learned_addr learned my addr 172.21.15.124:0/1817907082 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da14e8640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0c5d00 0x7f8d9c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da1ce9640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0c60e0 0x7f8d9c157410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 -- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0c5d00 msgr2=0x7f8d9c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0c5d00 0x7f8d9c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 -- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0c60e0 msgr2=0x7f8d9c157410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0c60e0 0x7f8d9c157410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 -- 172.21.15.124:0/1817907082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c158420 con 0x7f8d9c14ae90 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da0ce7640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c011960 tx=0x7f8d8c011d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00fa30 con 0x7f8d9c14ae90 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00fb90 con 0x7f8d9c14ae90 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c0cb9d0 con 0x7f8d9c14ae90 2023-12-13T23:55:20.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.868+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c014440 con 0x7f8d9c14ae90 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.869+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c157b30 con 0x7f8d9c14ae90 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.869+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c0cb830 con 0x7f8d9c14ae90 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.870+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8d8c01d400 con 0x7f8d9c14ae90 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.870+0000 7f8d91ffb640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 0x7f8d8006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.871+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d8c08a1b0 con 0x7f8d9c14ae90 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.871+0000 7f8da14e8640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 0x7f8d8006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.871+0000 7f8da14e8640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 0x7f8d8006d6b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004bc0 tx=0x7f8d98006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:20.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:20.875+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8d9c0cb830 con 0x7f8d9c14ae90 2023-12-13T23:55:21.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.047+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d9c0cb830 con 0x7f8d9c14ae90 2023-12-13T23:55:21.046 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.048+0000 7f8d91ffb640 1 -- 172.21.15.124:0/1817907082 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f8d9c0cb830 con 0x7f8d9c14ae90 2023-12-13T23:55:21.047 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:21.047 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 msgr2=0x7f8d8006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 0x7f8d8006d6b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8d98004bc0 tx=0x7f8d98006040 comp rx=0 tx=0).stop 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 msgr2=0x7f8d9c0b3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c011960 tx=0x7f8d8c011d60 comp rx=0 tx=0).stop 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 shutdown_connections 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8006b1f0 0x7f8d8006d6b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0c60e0 0x7f8d9c157410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c14ae90 0x7f8d9c0b3740 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 --2- 172.21.15.124:0/1817907082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0c5d00 0x7f8d9c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.050+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 >> 172.21.15.124:0/1817907082 conn(0x7f8d9c072020 msgr2=0x7f8d9c0c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.051+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 shutdown_connections 2023-12-13T23:55:21.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.051+0000 7f8da374c640 1 -- 172.21.15.124:0/1817907082 wait complete. 2023-12-13T23:55:21.059 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-13T23:55:21.060 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:21.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.161+0000 7fd6338b4640 1 Processor -- start 2023-12-13T23:55:21.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.161+0000 7fd6338b4640 1 -- start start 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd6338b4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 0x7fd62c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd6338b4640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c1406f0 con 0x7fd62c06abe0 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd6338b4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd62c143960 con 0x7fd62c140310 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd6338b4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c161e20 con 0x7fd62c1409d0 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd630e4f640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd62c06abe0 0x7fd62c06efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43560/0 (socket says 172.21.15.124:43560) 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd630e4f640 1 -- 172.21.15.124:0/1648948336 learned_addr learned my addr 172.21.15.124:0/1648948336 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd631e51640 1 --2- 172.21.15.124:0/1648948336 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 0x7fd62c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.162+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894722365 0 0) 0x7fd62c1406f0 con 0x7fd62c06abe0 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.163+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd618003520 con 0x7fd62c06abe0 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.163+0000 7fd631e51640 1 -- 172.21.15.124:0/1648948336 >> v1:172.21.15.203:6789/0 conn(0x7fd62c140310 legacy=0x7fd62c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.163+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917665475 0 0) 0x7fd618003520 con 0x7fd62c06abe0 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.163+0000 7fd631e51640 1 -- 172.21.15.124:0/1648948336 >> v1:172.21.15.124:6789/0 conn(0x7fd62c06abe0 legacy=0x7fd62c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.163+0000 7fd631e51640 1 -- 172.21.15.124:0/1648948336 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd62c142de0 con 0x7fd62c1409d0 2023-12-13T23:55:21.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd631e51640 1 --2- 172.21.15.124:0/1648948336 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 0x7fd62c1618e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097d0 tx=0x7fd62402b5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b582115106783bc9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c001670 con 0x7fd62c1409d0 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62402e050 con 0x7fd62c1409d0 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd623fff640 1 -- 172.21.15.124:0/1648948336 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624030710 con 0x7fd62c1409d0 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 -- 172.21.15.124:0/1648948336 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 msgr2=0x7fd62c1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 --2- 172.21.15.124:0/1648948336 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 0x7fd62c1618e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097d0 tx=0x7fd62402b5d0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 -- 172.21.15.124:0/1648948336 shutdown_connections 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 --2- 172.21.15.124:0/1648948336 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c1409d0 0x7fd62c1618e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 -- 172.21.15.124:0/1648948336 >> 172.21.15.124:0/1648948336 conn(0x7fd62c0b1e50 msgr2=0x7fd62c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 -- 172.21.15.124:0/1648948336 shutdown_connections 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.164+0000 7fd6338b4640 1 -- 172.21.15.124:0/1648948336 wait complete. 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.165+0000 7fd6338b4640 1 Processor -- start 2023-12-13T23:55:21.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 -- start start 2023-12-13T23:55:21.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd62c06abe0 0x7fd62c146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c140310 0x7fd62c1373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c143ea0 con 0x7fd62c1409d0 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c0ad600 con 0x7fd62c06abe0 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd6338b4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62c0ad8a0 con 0x7fd62c140310 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd631e51640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd631e51640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57034/0 (socket says 172.21.15.124:57034) 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.166+0000 7fd631e51640 1 -- 172.21.15.124:0/2946699926 learned_addr learned my addr 172.21.15.124:0/2946699926 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 -- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c140310 msgr2=0x7fd62c1373b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd630e4f640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c140310 0x7fd62c1373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c140310 0x7fd62c1373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631650640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd62c06abe0 0x7fd62c146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 -- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd62c06abe0 msgr2=0x7fd62c146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd62c06abe0 0x7fd62c146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 -- 172.21.15.124:0/2946699926 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624009400 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd631e51640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097d0 tx=0x7fd6240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624032810 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd624032970 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd62c0adb80 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624032d20 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.167+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd62c0ae140 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.168+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd62c0ac8e0 con 0x7fd62c1409d0 2023-12-13T23:55:21.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.169+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd624039070 con 0x7fd62c1409d0 2023-12-13T23:55:21.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.169+0000 7fd621ffb640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 0x7fd61006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.169+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6240abcc0 con 0x7fd62c1409d0 2023-12-13T23:55:21.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.169+0000 7fd631650640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 0x7fd61006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.170+0000 7fd631650640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 0x7fd61006d680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.173+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd624076b10 con 0x7fd62c1409d0 2023-12-13T23:55:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.343+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd62c161030 con 0x7fd62c1409d0 2023-12-13T23:55:21.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.344+0000 7fd621ffb640 1 -- 172.21.15.124:0/2946699926 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fd624037090 con 0x7fd62c1409d0 2023-12-13T23:55:21.343 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:21.343 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 msgr2=0x7fd61006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 0x7fd61006d680 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd628007fd0 tx=0x7fd628009040 comp rx=0 tx=0).stop 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 msgr2=0x7fd62c144a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd6240097d0 tx=0x7fd6240046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 shutdown_connections 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd61006b1c0 0x7fd61006d680 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd62c1409d0 0x7fd62c144a40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd62c140310 0x7fd62c1373b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 --2- 172.21.15.124:0/2946699926 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd62c06abe0 0x7fd62c146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.346+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 >> 172.21.15.124:0/2946699926 conn(0x7fd62c0b1e50 msgr2=0x7fd62c073490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.347+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 shutdown_connections 2023-12-13T23:55:21.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.347+0000 7fd6338b4640 1 -- 172.21.15.124:0/2946699926 wait complete. 2023-12-13T23:55:21.356 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-13T23:55:21.356 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-13T23:55:21.357 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-13T23:55:21.357 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-13T23:55:21.358 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-13T23:55:21.358 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-13T23:55:21.358 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-13T23:55:21.359 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-13T23:55:21.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.530+0000 7f9371cad640 1 Processor -- start 2023-12-13T23:55:21.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.530+0000 7f47ff0ff640 1 Processor -- start 2023-12-13T23:55:21.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.530+0000 7fd248878640 1 Processor -- start 2023-12-13T23:55:21.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7fbf29533640 1 Processor -- start 2023-12-13T23:55:21.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7fbf29533640 1 -- start start 2023-12-13T23:55:21.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.530+0000 7f9371cad640 1 -- start start 2023-12-13T23:55:21.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7fd248878640 1 -- start start 2023-12-13T23:55:21.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7f9371cad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c159da0 0x7f936c0cfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.530+0000 7f47ff0ff640 1 -- start start 2023-12-13T23:55:21.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7f9371cad640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936c0d0640 con 0x7f936c15e160 2023-12-13T23:55:21.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.531+0000 7f9371cad640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f936c15a180 con 0x7f936c0d01e0 2023-12-13T23:55:21.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f9371cad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c15e5d0 con 0x7f936c159da0 2023-12-13T23:55:21.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f93714ac640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f936c15e160 0x7f936c06d4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43566/0 (socket says 172.21.15.124:43566) 2023-12-13T23:55:21.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f76d5f1d640 1 Processor -- start 2023-12-13T23:55:21.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f93714ac640 1 -- 172.21.15.124:0/3611709026 learned_addr learned my addr 172.21.15.124:0/3611709026 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f76d5f1d640 1 -- start start 2023-12-13T23:55:21.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf29533640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf2406a4d0 0x7fbf240b4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f9370cab640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c159da0 0x7f936c0cfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd248878640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 0x7fd244071c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf29533640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbf240ad250 0x7fbf2406a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf29533640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b3bb0 con 0x7fbf240ad6b0 2023-12-13T23:55:21.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf29533640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240acb70 con 0x7fbf240ad250 2023-12-13T23:55:21.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf29533640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b2cc0 con 0x7fbf2406a4d0 2023-12-13T23:55:21.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47ff0ff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f47f81607d0 0x7f47f80cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47ff0ff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47ff0ff640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47f80cebf0 con 0x7f47f80cff20 2023-12-13T23:55:21.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47ff0ff640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47f80d0390 con 0x7f47f81607d0 2023-12-13T23:55:21.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47ff0ff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80cee90 con 0x7f47f80cf310 2023-12-13T23:55:21.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fba1e34e640 1 Processor -- start 2023-12-13T23:55:21.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf22ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf2406a4d0 0x7fbf240b4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf22ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf2406a4d0 0x7fbf240b4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59870/0 (socket says 172.21.15.124:59870) 2023-12-13T23:55:21.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47fd8fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf22ffd640 1 -- 172.21.15.124:0/4121815761 learned_addr learned my addr 172.21.15.124:0/4121815761 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fbf1a7fc640 1 --2- 172.21.15.124:0/4121815761 >> v2:172.21.15.203:3300/0 conn(0x7fbf240ad250 0x7fbf2406a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47fd8fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59884/0 (socket says 172.21.15.124:59884) 2023-12-13T23:55:21.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fba1e34e640 1 -- start start 2023-12-13T23:55:21.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47fd8fc640 1 -- 172.21.15.124:0/3374500096 learned_addr learned my addr 172.21.15.124:0/3374500096 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7f47fe0fd640 1 --2- 172.21.15.124:0/3374500096 >> v2:172.21.15.124:3300/0 conn(0x7f47f81607d0 0x7f47f80cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd242ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd2440721a0 0x7fd24406a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33006/0 (socket says 172.21.15.124:33006) 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd2437fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 0x7fd244071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd2437fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 0x7fd244071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51394/0 (socket says 172.21.15.124:51394) 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd248878640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd2440cfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd242ffd640 1 -- 172.21.15.124:0/3513534757 learned_addr learned my addr 172.21.15.124:0/3513534757 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd243fff640 1 --2- 172.21.15.124:0/3513534757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd2440cfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2440cee20 con 0x7fd2440721a0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.532+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24406f110 con 0x7fd2440b4a00 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2440cf100 con 0x7fd24406abe0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd2427fc640 1 -- 172.21.15.124:0/3513534757 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560966502 0 0) 0x7fd2440cee20 con 0x7fd2440721a0 2023-12-13T23:55:21.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd2427fc640 1 -- 172.21.15.124:0/3513534757 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd22c003520 con 0x7fd2440721a0 2023-12-13T23:55:21.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740851536 0 0) 0x7fbf240b3bb0 con 0x7fbf240ad6b0 2023-12-13T23:55:21.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf10003520 con 0x7fbf240ad6b0 2023-12-13T23:55:21.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76d5f1d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f76d0161840 0x7f76d00b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd0fb640 1 -- 172.21.15.124:0/3374500096 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1746925561 0 0) 0x7f47f80cebf0 con 0x7f47f80cff20 2023-12-13T23:55:21.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd0fb640 1 -- 172.21.15.124:0/3374500096 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e4003520 con 0x7f47f80cff20 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd243fff640 1 -- 172.21.15.124:0/3513534757 >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 msgr2=0x7fd244071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd243fff640 1 --2- 172.21.15.124:0/3513534757 >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 0x7fd244071c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd243fff640 1 -- 172.21.15.124:0/3513534757 >> v1:172.21.15.203:6789/0 conn(0x7fd2440721a0 legacy=0x7fd24406a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fd243fff640 1 -- 172.21.15.124:0/3513534757 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2440cf3e0 con 0x7fd24406abe0 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd8fc640 1 -- 172.21.15.124:0/3374500096 >> v2:172.21.15.124:3300/0 conn(0x7f47f81607d0 msgr2=0x7f47f80cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd8fc640 1 --2- 172.21.15.124:0/3374500096 >> v2:172.21.15.124:3300/0 conn(0x7f47f81607d0 0x7f47f80cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd8fc640 1 -- 172.21.15.124:0/3374500096 >> v1:172.21.15.124:6789/0 conn(0x7f47f80cff20 legacy=0x7f47f80d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f47fd8fc640 1 -- 172.21.15.124:0/3374500096 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f80b2990 con 0x7f47f80cf310 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76d5f1d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f76d00b7bf0 0x7f76d0165d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76d5f1d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76d00d20b0 con 0x7f76d0161c20 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76d5f1d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00b77d0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76d5f1d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76d00b8040 con 0x7f76d0161840 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819744162 0 0) 0x7f936c15a180 con 0x7f936c0d01e0 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f935c003520 con 0x7f936c0d01e0 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249433240 0 0) 0x7f936c0d0640 con 0x7f936c15e160 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f936c15a180 con 0x7f936c15e160 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 996279149 0 0) 0x7f935c003520 con 0x7f936c0d01e0 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c159da0 msgr2=0x7f936c0cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c159da0 0x7f936c0cfca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 >> v1:172.21.15.124:6789/0 conn(0x7f936c15e160 legacy=0x7f936c06d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c15e7f0 con 0x7f936c0d01e0 2023-12-13T23:55:21.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f9358001670 con 0x7f936c0d01e0 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 >> v1:172.21.15.203:6789/0 conn(0x7f936c0d01e0 legacy=0x7f936c15da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1542921404 0 0) 0x7fbf10003520 con 0x7fbf240ad6b0 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf2406a4d0 msgr2=0x7fbf240b4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf2406a4d0 0x7fbf240b4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 >> v2:172.21.15.203:3300/0 conn(0x7fbf240ad250 msgr2=0x7fbf2406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 --2- 172.21.15.124:0/4121815761 >> v2:172.21.15.203:3300/0 conn(0x7fbf240ad250 0x7fbf2406a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b2fa0 con 0x7fbf240ad6b0 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fbf1c0019f0 con 0x7fbf240ad6b0 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 >> v1:172.21.15.203:6789/0 conn(0x7fbf240ad6b0 legacy=0x7fbf2406ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf10003970 0x7fbf10003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf10004380 0x7fbf10004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf10004f80 0x7fbf1000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240acb70 con 0x7fbf10004f80 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b2cc0 con 0x7fbf10004380 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1000bf10 con 0x7fbf10003970 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f1cddfdc640 1 Processor -- start 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba1004fcb0 0x7fba100540a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fba100549a0 0x7fba10050090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba10052ab0 0x7fba10052f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba100505d0 con 0x7fba100549a0 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba10007030 con 0x7fba1004fcb0 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1e34e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba10007310 con 0x7fba10052ab0 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f58671fb640 1 Processor -- start 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76cffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f76d00b7bf0 0x7f76d0165d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76cffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f76d00b7bf0 0x7f76d0165d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57056/0 (socket says 172.21.15.124:57056) 2023-12-13T23:55:21.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f76cffff640 1 -- 172.21.15.124:0/2698153324 learned_addr learned my addr 172.21.15.124:0/2698153324 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f935c003b00 0x7f935c003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c0047c0 0x7f935c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f935c005180 0x7f935c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c15e5d0 con 0x7f935c005180 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f935c00bf70 con 0x7f935c0047c0 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f935c00c130 con 0x7f935c003b00 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f76cffff640 1 -- 172.21.15.124:0/2698153324 >> v2:172.21.15.203:3300/0 conn(0x7f76d0161840 msgr2=0x7f76d00b2330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-13T23:55:21.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f76cffff640 1 --2- 172.21.15.124:0/2698153324 >> v2:172.21.15.203:3300/0 conn(0x7f76d0161840 0x7f76d00b2330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f76cffff640 1 -- 172.21.15.124:0/2698153324 >> v1:172.21.15.203:6789/0 conn(0x7f76d0161c20 legacy=0x7f76d0165600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f76cffff640 1 -- 172.21.15.124:0/2698153324 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d00b73f0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f58671fb640 1 -- start start 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fd243fff640 1 --2- 172.21.15.124:0/3513534757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd2440cfbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009d90 tx=0x7fd23c02be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37e44229f5d9eb5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf1a7fc640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf10004380 0x7fbf10004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf22ffd640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf10003970 0x7fbf10003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fbf237fe640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf10004f80 0x7fbf1000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1cb4b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fba100549a0 0x7fba10050090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1cb4b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fba100549a0 0x7fba10050090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57086/0 (socket says 172.21.15.124:57086) 2023-12-13T23:55:21.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7fba1cb4b640 1 -- 172.21.15.124:0/263661390 learned_addr learned my addr 172.21.15.124:0/263661390 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1d34c640 1 --2- 172.21.15.124:0/263661390 >> v2:172.21.15.203:3300/0 conn(0x7fba1004fcb0 0x7fba100540a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fd2427fc640 1 -- 172.21.15.124:0/3513534757 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c02cc00 con 0x7fd24406abe0 2023-12-13T23:55:21.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fd2427fc640 1 -- 172.21.15.124:0/3513534757 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23c02cd60 con 0x7fd24406abe0 2023-12-13T23:55:21.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fd2427fc640 1 -- 172.21.15.124:0/3513534757 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23c034660 con 0x7fd24406abe0 2023-12-13T23:55:21.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.533+0000 7f1cddfdc640 1 -- start start 2023-12-13T23:55:21.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f936bfff640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c0047c0 0x7f935c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cddfdc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1cd80d3cb0 0x7f1cd80d1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cddfdc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cd80d1fa0 0x7f1cd80cfb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cddfdc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd80d0cc0 con 0x7f1cd815d1e0 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cddfdc640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd80cef90 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cddfdc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd80d0f80 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58671fb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f58580e3090 0x7f5858057d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cd77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1cd815d1e0 0x7f1cd80ce760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33046/0 (socket says 172.21.15.124:33046) 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cd77fe640 1 -- 172.21.15.124:0/1902801068 learned_addr learned my addr 172.21.15.124:0/1902801068 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f47fd8fc640 1 --2- 172.21.15.124:0/3374500096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009e50 tx=0x7f47f002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=89e7ef018c5cbde server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58671fb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 0x7f58580068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58671fb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58580582a0 con 0x7f585803e810 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58671fb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5858006170 con 0x7f58580e3090 2023-12-13T23:55:21.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58671fb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5858058540 con 0x7f58580542d0 2023-12-13T23:55:21.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f47fd0fb640 1 -- 172.21.15.124:0/3374500096 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f002cb00 con 0x7f47f80cf310 2023-12-13T23:55:21.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1cb4b640 1 -- 172.21.15.124:0/263661390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba10052ab0 msgr2=0x7fba10052f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:55:21.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f9370cab640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f935c003b00 0x7f935c003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.534+0000 7f93714ac640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f935c005180 0x7f935c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fbf237fe640 1 -- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf10003970 msgr2=0x7fbf10003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fbf237fe640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf10003970 0x7fbf10003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/263661390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba10052ab0 0x7fba10052f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1cb4b640 1 -- 172.21.15.124:0/263661390 >> v2:172.21.15.203:3300/0 conn(0x7fba1004fcb0 msgr2=0x7fba100540a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f47fd0fb640 1 -- 172.21.15.124:0/3374500096 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f002cc60 con 0x7f47f80cf310 2023-12-13T23:55:21.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fbf237fe640 1 -- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf10004380 msgr2=0x7fbf10004860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fbf237fe640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf10004380 0x7fbf10004860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/263661390 >> v2:172.21.15.203:3300/0 conn(0x7fba1004fcb0 0x7fba100540a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58659f8640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f585803e810 0x7f5858053bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43574/0 (socket says 172.21.15.124:43574) 2023-12-13T23:55:21.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f58659f8640 1 -- 172.21.15.124:0/606054333 learned_addr learned my addr 172.21.15.124:0/606054333 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fba1cb4b640 1 -- 172.21.15.124:0/263661390 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100075f0 con 0x7fba100549a0 2023-12-13T23:55:21.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7fbf237fe640 1 -- 172.21.15.124:0/4121815761 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf10003520 con 0x7fbf10004f80 2023-12-13T23:55:21.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/1902801068 >> v2:172.21.15.203:3300/0 conn(0x7f1cd80d3cb0 0x7f1cd80d1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f1cd7fff640 1 --2- 172.21.15.124:0/1902801068 >> v2:172.21.15.124:3300/0 conn(0x7f1cd80d1fa0 0x7f1cd80cfb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f76cffff640 1 --2- 172.21.15.124:0/2698153324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d00b7bf0 0x7f76d0165d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f76d00743e0 tx=0x7f76c802bf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb1e126e7a23b329 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f76ce7fc640 1 -- 172.21.15.124:0/2698153324 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c802caa0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206947933 0 0) 0x7f58580582a0 con 0x7f585803e810 2023-12-13T23:55:21.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f584c003520 con 0x7f585803e810 2023-12-13T23:55:21.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f76ce7fc640 1 -- 172.21.15.124:0/2698153324 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c802cc00 con 0x7f76d00b7bf0 2023-12-13T23:55:21.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f76ce7fc640 1 -- 172.21.15.124:0/2698153324 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c80343f0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f93714ac640 1 -- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f935c003b00 msgr2=0x7f935c003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f93714ac640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f935c003b00 0x7f935c003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f93714ac640 1 -- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c0047c0 msgr2=0x7f935c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3374500096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 msgr2=0x7f47f80cf9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3374500096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009e50 tx=0x7f47f002bf60 comp rx=0 tx=0).stop 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 msgr2=0x7fd2440cfbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 --2- 172.21.15.124:0/3513534757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd2440cfbe0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009d90 tx=0x7fd23c02be50 comp rx=0 tx=0).stop 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 shutdown_connections 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 --2- 172.21.15.124:0/3513534757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd2440cfbe0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f93714ac640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c0047c0 0x7f935c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.535+0000 7f93714ac640 1 -- 172.21.15.124:0/3611709026 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c15e7f0 con 0x7f935c005180 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 --2- 172.21.15.124:0/3513534757 >> v2:172.21.15.203:3300/0 conn(0x7fd2440b4a00 0x7fd244071c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 >> 172.21.15.124:0/3513534757 conn(0x7fd2440b1a60 msgr2=0x7fd2440b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 shutdown_connections 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7fd248878640 1 -- 172.21.15.124:0/3513534757 wait complete. 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f1cd6ffd640 1 -- 172.21.15.124:0/1902801068 >> v2:172.21.15.124:3300/0 conn(0x7f1cd80d1fa0 msgr2=0x7f1cd80cfb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/1902801068 >> v2:172.21.15.124:3300/0 conn(0x7f1cd80d1fa0 0x7f1cd80cfb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f1cd6ffd640 1 -- 172.21.15.124:0/1902801068 >> v1:172.21.15.203:6789/0 conn(0x7f1cd815d1e0 legacy=0x7f1cd80ce760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f1cd6ffd640 1 -- 172.21.15.124:0/1902801068 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd80d1260 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934934492 0 0) 0x7f584c003520 con 0x7f585803e810 2023-12-13T23:55:21.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 msgr2=0x7f58580068a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 0x7f58580068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 >> v2:172.21.15.203:3300/0 conn(0x7f58580e3090 msgr2=0x7f5858057d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3374500096 shutdown_connections 2023-12-13T23:55:21.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3374500096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cf310 0x7f47f80cf9e0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009e50 tx=0x7f47f002bf60 comp rx=0 tx=0).stop 2023-12-13T23:55:21.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3374500096 >> v2:172.21.15.124:3300/0 conn(0x7f47f81607d0 0x7f47f80cdb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3374500096 >> 172.21.15.124:0/3374500096 conn(0x7f47f80d2fd0 msgr2=0x7f47f80d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58661f9640 1 --2- 172.21.15.124:0/606054333 >> v2:172.21.15.203:3300/0 conn(0x7f58580e3090 0x7f5858057d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.536+0000 7f58651f7640 1 --2- 172.21.15.124:0/606054333 >> v2:172.21.15.203:3300/0 conn(0x7f58580e3090 0x7f5858057d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858058820 con 0x7f585803e810 2023-12-13T23:55:21.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f585c001670 con 0x7f585803e810 2023-12-13T23:55:21.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 >> v1:172.21.15.124:6789/0 conn(0x7f585803e810 legacy=0x7f5858053bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f93714ac640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f935c005180 0x7f935c00b980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9364015ba0 tx=0x7f935c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ef95a69cd62e33 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/263661390 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba100549a0 0x7fba10050090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba180730e0 tx=0x7fba180a5e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8626aba491365cc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f584c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f584c0047c0 0x7f584c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9364036480 con 0x7f935c005180 2023-12-13T23:55:21.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 0x7f584c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/1902801068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80d1a60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1cd808b000 tx=0x7f1cc802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c989893da313c5c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5858006170 con 0x7f584c003970 2023-12-13T23:55:21.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5858058540 con 0x7f584c0047c0 2023-12-13T23:55:21.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584c005620 con 0x7f58580542d0 2023-12-13T23:55:21.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7fbf237fe640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf10004f80 0x7fbf1000b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c004390 tx=0x7fbf100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f22576472a70852 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f584c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58669fa640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 0x7f584c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 -- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 msgr2=0x7f584c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580542d0 0x7f584c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 -- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f584c0047c0 msgr2=0x7f584c004c40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f584c0047c0 0x7f584c004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f58661f9640 1 -- 172.21.15.124:0/606054333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858058820 con 0x7f584c003970 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 -- 172.21.15.124:0/2698153324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d00b7bf0 msgr2=0x7f76d0165d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/2698153324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d00b7bf0 0x7f76d0165d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f76d00743e0 tx=0x7f76c802bf00 comp rx=0 tx=0).stop 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 -- 172.21.15.124:0/2698153324 shutdown_connections 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/2698153324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d00b7bf0 0x7f76d0165d10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/2698153324 >> v2:172.21.15.203:3300/0 conn(0x7f76d0161840 0x7f76d00b2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 -- 172.21.15.124:0/2698153324 >> 172.21.15.124:0/2698153324 conn(0x7f76d00d31f0 msgr2=0x7f76d00d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f1cd67fc640 1 -- 172.21.15.124:0/1902801068 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8030710 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f1cd67fc640 1 -- 172.21.15.124:0/1902801068 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc802e050 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3374500096 shutdown_connections 2023-12-13T23:55:21.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f1cd67fc640 1 -- 172.21.15.124:0/1902801068 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8030a90 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3374500096 wait complete. 2023-12-13T23:55:21.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c004bc0 con 0x7fbf10004f80 2023-12-13T23:55:21.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7fba0ffff640 1 -- 172.21.15.124:0/263661390 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180a6cf0 con 0x7fba100549a0 2023-12-13T23:55:21.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c02f070 con 0x7fbf10004f80 2023-12-13T23:55:21.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fba0ffff640 1 -- 172.21.15.124:0/263661390 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180a6e90 con 0x7fba100549a0 2023-12-13T23:55:21.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fba0ffff640 1 -- 172.21.15.124:0/263661390 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba180ae640 con 0x7fba100549a0 2023-12-13T23:55:21.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fbf227fc640 1 -- 172.21.15.124:0/4121815761 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1c006070 con 0x7fbf10004f80 2023-12-13T23:55:21.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fd248878640 1 Processor -- start 2023-12-13T23:55:21.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f76d5f1d640 1 -- 172.21.15.124:0/2698153324 shutdown_connections 2023-12-13T23:55:21.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7fd248878640 1 -- start start 2023-12-13T23:55:21.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93640365e0 con 0x7f935c005180 2023-12-13T23:55:21.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.537+0000 7f936b7fe640 1 -- 172.21.15.124:0/3611709026 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93640369b0 con 0x7f935c005180 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f9371cad640 1 -- 172.21.15.124:0/3611709026 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f935c005180 msgr2=0x7f935c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f9371cad640 1 --2- 172.21.15.124:0/3611709026 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f935c005180 0x7f935c00b980 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9364015ba0 tx=0x7f935c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f93714ac640 1 -- 172.21.15.124:0/3611709026 reap_dead start 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f9371cad640 1 -- 172.21.15.124:0/3611709026 shutdown_connections 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f9371cad640 1 -- 172.21.15.124:0/3611709026 >> 172.21.15.124:0/3611709026 conn(0x7f936c0b1a60 msgr2=0x7f936c0b1e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 -- 172.21.15.124:0/1902801068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 msgr2=0x7f1cd80d1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/1902801068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80d1a60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1cd808b000 tx=0x7f1cc802b710 comp rx=0 tx=0).stop 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 -- 172.21.15.124:0/1902801068 shutdown_connections 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/1902801068 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80d1a60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/1902801068 >> v2:172.21.15.124:3300/0 conn(0x7f1cd80d1fa0 0x7f1cd80cfb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f1cddfdc640 1 -- 172.21.15.124:0/1902801068 >> 172.21.15.124:0/1902801068 conn(0x7f1cd809c300 msgr2=0x7f1cd80d30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 -- 172.21.15.124:0/263661390 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba100549a0 msgr2=0x7fba10050090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 --2- 172.21.15.124:0/263661390 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba100549a0 0x7fba10050090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fba180730e0 tx=0x7fba180a5e80 comp rx=0 tx=0).stop 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f58661f9640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f584c003d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5860069d60 tx=0x7f584c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd740564424f1e23 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600a6ae0 con 0x7f584c003970 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58600a6c40 con 0x7f584c003970 2023-12-13T23:55:21.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.538+0000 7f76d5f1d640 1 -- 172.21.15.124:0/2698153324 wait complete. 2023-12-13T23:55:21.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 -- 172.21.15.124:0/263661390 shutdown_connections 2023-12-13T23:55:21.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 --2- 172.21.15.124:0/263661390 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba100549a0 0x7fba10050090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 --2- 172.21.15.124:0/263661390 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba10052ab0 0x7fba10052f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 --2- 172.21.15.124:0/263661390 >> v2:172.21.15.203:3300/0 conn(0x7fba1004fcb0 0x7fba100540a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 -- 172.21.15.124:0/4121815761 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf10004f80 msgr2=0x7fbf1000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 -- 172.21.15.124:0/263661390 >> 172.21.15.124:0/263661390 conn(0x7fba10054ee0 msgr2=0x7fba100552f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 --2- 172.21.15.124:0/4121815761 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf10004f80 0x7fbf1000b980 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c004390 tx=0x7fbf100057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf237fe640 1 -- 172.21.15.124:0/4121815761 reap_dead start 2023-12-13T23:55:21.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 -- 172.21.15.124:0/4121815761 shutdown_connections 2023-12-13T23:55:21.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 -- 172.21.15.124:0/4121815761 >> 172.21.15.124:0/4121815761 conn(0x7fbf240cf410 msgr2=0x7fbf240cf820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f9371cad640 1 -- 172.21.15.124:0/3611709026 shutdown_connections 2023-12-13T23:55:21.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f9371cad640 1 -- 172.21.15.124:0/3611709026 wait complete. 2023-12-13T23:55:21.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 -- 172.21.15.124:0/263661390 shutdown_connections 2023-12-13T23:55:21.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f58651f7640 1 -- 172.21.15.124:0/606054333 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58600a6ae0 con 0x7f584c003970 2023-12-13T23:55:21.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fba1e34e640 1 -- 172.21.15.124:0/263661390 wait complete. 2023-12-13T23:55:21.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f1cddfdc640 1 -- 172.21.15.124:0/1902801068 shutdown_connections 2023-12-13T23:55:21.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f1cddfdc640 1 -- 172.21.15.124:0/1902801068 wait complete. 2023-12-13T23:55:21.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fd248878640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd244164640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f47ff0ff640 1 Processor -- start 2023-12-13T23:55:21.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7f47ff0ff640 1 -- start start 2023-12-13T23:55:21.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 -- 172.21.15.124:0/4121815761 shutdown_connections 2023-12-13T23:55:21.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f58671fb640 1 -- 172.21.15.124:0/606054333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 msgr2=0x7f584c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7fd248878640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 0x7fd24414f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f58671fb640 1 --2- 172.21.15.124:0/606054333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f584c003d90 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f5860069d60 tx=0x7f584c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f58669fa640 1 -- 172.21.15.124:0/606054333 reap_dead start 2023-12-13T23:55:21.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f76d5f1d640 1 Processor -- start 2023-12-13T23:55:21.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7fd248878640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2440b4a00 0x7fd24409b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7fd248878640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd244099d80 con 0x7fd2440721a0 2023-12-13T23:55:21.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7fd248878640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24414d730 con 0x7fd2440b4a00 2023-12-13T23:55:21.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7fd248878640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24414da10 con 0x7fd24406abe0 2023-12-13T23:55:21.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f58671fb640 1 -- 172.21.15.124:0/606054333 shutdown_connections 2023-12-13T23:55:21.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f58671fb640 1 -- 172.21.15.124:0/606054333 >> 172.21.15.124:0/606054333 conn(0x7f58580559b0 msgr2=0x7f5858055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.540+0000 7f76d5f1d640 1 -- start start 2023-12-13T23:55:21.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.539+0000 7fbf29533640 1 -- 172.21.15.124:0/4121815761 wait complete. 2023-12-13T23:55:21.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f9371cad640 1 Processor -- start 2023-12-13T23:55:21.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47ff0ff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 0x7f47f814f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f9371cad640 1 -- start start 2023-12-13T23:55:21.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47ff0ff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47f81607d0 0x7f47f80b6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fd2437fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd244164640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fba1e34e640 1 Processor -- start 2023-12-13T23:55:21.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f1cddfdc640 1 Processor -- start 2023-12-13T23:55:21.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fd242ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 0x7fd24414f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fba1e34e640 1 -- start start 2023-12-13T23:55:21.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47fe0fd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 0x7f47f814f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fd242ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 0x7fd24414f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57114/0 (socket says 172.21.15.124:57114) 2023-12-13T23:55:21.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fd242ffd640 1 -- 172.21.15.124:0/2279747603 learned_addr learned my addr 172.21.15.124:0/2279747603 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47fe0fd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 0x7f47f814f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59954/0 (socket says 172.21.15.124:59954) 2023-12-13T23:55:21.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fd2437fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd244164640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59948/0 (socket says 172.21.15.124:59948) 2023-12-13T23:55:21.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47fe0fd640 1 -- 172.21.15.124:0/3868715985 learned_addr learned my addr 172.21.15.124:0/3868715985 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7fbf29533640 1 Processor -- start 2023-12-13T23:55:21.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47f80b6680 0x7f47f806f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47fd8fc640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47f81607d0 0x7f47f80b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f47fe8fe640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47f80b6680 0x7f47f806f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 0x7f76d0153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76d0161840 0x7f76d00b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80b7f10 con 0x7f47f80b6680 2023-12-13T23:55:21.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d0161c20 0x7f76d00b1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fd242ffd640 1 -- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 msgr2=0x7fd244164640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fd242ffd640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd24406abe0 0x7fd244164640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fd242ffd640 1 -- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2440b4a00 msgr2=0x7fd24409b360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fd242ffd640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2440b4a00 0x7fd24409b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fd242ffd640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23c009a70 con 0x7fd2440721a0 2023-12-13T23:55:21.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 0x7f936c06a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76cf7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 0x7f76d0153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0164a50 con 0x7f76d0161c20 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d0163dd0 con 0x7f76d0161840 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76d5f1d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76d01640b0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76cf7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 0x7f76d0153f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59958/0 (socket says 172.21.15.124:59958) 2023-12-13T23:55:21.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.541+0000 7f1cddfdc640 1 -- start start 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76cf7fe640 1 -- 172.21.15.124:0/812908571 learned_addr learned my addr 172.21.15.124:0/812908571 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 0x7f936c0adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c147300 0x7f936c147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c1575d0 con 0x7f936c0ad6c0 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c147cc0 con 0x7f935c005180 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9371cad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936c0b01e0 con 0x7f936c147300 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f936bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 0x7f936c0adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9370cab640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 0x7f936c06a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9370cab640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 0x7f936c06a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51474/0 (socket says 172.21.15.124:51474) 2023-12-13T23:55:21.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f936bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 0x7f936c0adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57150/0 (socket says 172.21.15.124:57150) 2023-12-13T23:55:21.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f9370cab640 1 -- 172.21.15.124:0/4195712959 learned_addr learned my addr 172.21.15.124:0/4195712959 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76cffff640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d0161c20 0x7f76d00b1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f76ceffd640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76d0161840 0x7f76d00b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80b6e70 con 0x7f47f81607d0 2023-12-13T23:55:21.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f80b7150 con 0x7f47f80cff20 2023-12-13T23:55:21.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 0x7fba100d7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba10052ab0 0x7fba100d60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f93714ac640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c147300 0x7f936c147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba100549a0 0x7fba100d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100e08a0 con 0x7fba1004fcb0 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100dfc20 con 0x7fba10052ab0 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fba1e34e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba100dff00 con 0x7fba100549a0 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f1cddfdc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 0x7f1cd815c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7f1cddfdc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cddfdc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd815d1e0 0x7f1cd80b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 -- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c147300 msgr2=0x7f936c147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f936c147300 0x7f936c147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 -- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 msgr2=0x7f936c0adb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 0x7f936c0adb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936c0b04c0 con 0x7f935c005180 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f936bfff640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f936c0ad6c0 0x7f936c0adb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 0x7f1cd815c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cddfdc640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd81f7940 con 0x7f1cd815d1e0 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cddfdc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd815c9a0 con 0x7f1cd80d3cb0 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 0x7f1cd815c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59974/0 (socket says 172.21.15.124:59974) 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 -- 172.21.15.124:0/3704653029 learned_addr learned my addr 172.21.15.124:0/3704653029 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cddfdc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd81f7050 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 0x7fba100d7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1db4d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba100549a0 0x7fba100d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 0x7fba100d7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57154/0 (socket says 172.21.15.124:57154) 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 -- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76d0161840 msgr2=0x7f76d00b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76d0161840 0x7f76d00b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 -- 172.21.15.124:0/812908571 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d0161c20 msgr2=0x7f76d00b1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f76d0161c20 0x7f76d00b1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8009b30 con 0x7f76d00b7bf0 2023-12-13T23:55:21.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.542+0000 7fbf29533640 1 -- start start 2023-12-13T23:55:21.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 -- 172.21.15.124:0/637841442 learned_addr learned my addr 172.21.15.124:0/637841442 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 0x7fbf24160010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf241e59f0 0x7fbf2415f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf240b0040 0x7fbf240b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fd242ffd640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 0x7fd24414f650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c390 tx=0x7fd23400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd7fff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd815d1e0 0x7f1cd80b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf22ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 0x7fbf24160010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf22ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 0x7fbf24160010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57180/0 (socket says 172.21.15.124:57180) 2023-12-13T23:55:21.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f76cf7fe640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 0x7f76d0153f20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76c000b480 tx=0x7f76c000b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9370cab640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 0x7f936c06a740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93600115d0 tx=0x7f9360011a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240b0a10 con 0x7fbf24149e50 2023-12-13T23:55:21.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000f8f0 con 0x7f935c005180 2023-12-13T23:55:21.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba10052ab0 0x7fba100d60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd234005ce0 con 0x7fd2440721a0 2023-12-13T23:55:21.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd234005e40 con 0x7fd2440721a0 2023-12-13T23:55:21.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2340072d0 con 0x7fd2440721a0 2023-12-13T23:55:21.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fd248878640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24414dcf0 con 0x7fd2440721a0 2023-12-13T23:55:21.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 -- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba100549a0 msgr2=0x7fba100d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f58671fb640 1 -- 172.21.15.124:0/606054333 shutdown_connections 2023-12-13T23:55:21.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936000fa50 con 0x7f935c005180 2023-12-13T23:55:21.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba100549a0 0x7fba100d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9371cad640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c0b07a0 con 0x7f935c005180 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 -- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba10052ab0 msgr2=0x7fba100d60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba10052ab0 0x7fba100d60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe0fd640 1 -- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47f81607d0 msgr2=0x7f47f80b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe0fd640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f47f81607d0 0x7f47f80b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe0fd640 1 -- 172.21.15.124:0/3868715985 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47f80b6680 msgr2=0x7f47f806f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fba1d34c640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1804ed60 con 0x7fba1004fcb0 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe0fd640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47f80b6680 0x7f47f806f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe0fd640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f0009b30 con 0x7f47f80cff20 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f47fe8fe640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47f80b6680 0x7f47f806f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f9371cad640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c0afeb0 con 0x7f935c005180 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2415e090 con 0x7fbf240b0040 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf29533640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf24160550 con 0x7fbf241e59f0 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf1a7fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf241e59f0 0x7fbf2415f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf1a7fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf241e59f0 0x7fbf2415f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:59984/0 (socket says 172.21.15.124:59984) 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7fbf1a7fc640 1 -- 172.21.15.124:0/3881174112 learned_addr learned my addr 172.21.15.124:0/3881174112 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf237fe640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf240b0040 0x7fbf240b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 -- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 msgr2=0x7f1cd80cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd80d3cb0 0x7f1cd80cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 -- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd815d1e0 msgr2=0x7f1cd80b32b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd815d1e0 0x7f1cd80b32b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f1cd77fe640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc8009400 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f1cd7fff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd815d1e0 0x7f1cd80b32b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:21.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0010870 con 0x7f76d00b7bf0 2023-12-13T23:55:21.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76c00109d0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47fe0fd640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 0x7f47f814f200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47f400d830 tx=0x7f47f400dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fba1d34c640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 0x7fba100d7250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba1400e9b0 tx=0x7fba1400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.543+0000 7f58671fb640 1 -- 172.21.15.124:0/606054333 wait complete. 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 -- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf241e59f0 msgr2=0x7fbf2415f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf241e59f0 0x7fbf2415f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 -- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf240b0040 msgr2=0x7fbf240b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf240b0040 0x7fbf240b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400be10 con 0x7f47f80cff20 2023-12-13T23:55:21.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2415e2a0 con 0x7fbf24149e50 2023-12-13T23:55:21.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf237fe640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf240b0040 0x7fbf240b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:21.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f40098b0 con 0x7f47f80cff20 2023-12-13T23:55:21.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7fbf22ffd640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 0x7fbf24160010 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c011510 tx=0x7fbf0c0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f400be10 con 0x7f47f80cff20 2023-12-13T23:55:21.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f80b7430 con 0x7f47f80cff20 2023-12-13T23:55:21.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f80b79f0 con 0x7f47f80cff20 2023-12-13T23:55:21.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fd248878640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd24414e2b0 con 0x7fd2440721a0 2023-12-13T23:55:21.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936000fe00 con 0x7f935c005180 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.544+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76c0010d80 con 0x7f76d00b7bf0 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d0164390 con 0x7f76d00b7bf0 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d0154460 con 0x7f76d00b7bf0 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00c070 con 0x7fbf24149e50 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c015930 con 0x7fbf24149e50 2023-12-13T23:55:21.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c015cc0 con 0x7fbf24149e50 2023-12-13T23:55:21.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7fd248878640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2440b4de0 con 0x7fd2440721a0 2023-12-13T23:55:21.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9360015460 con 0x7f935c005180 2023-12-13T23:55:21.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2415e580 con 0x7fbf24149e50 2023-12-13T23:55:21.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.545+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2415eb40 con 0x7fbf24149e50 2023-12-13T23:55:21.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f1cd77fe640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 0x7f1cd815c460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000caf0 tx=0x7f1cd0002080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f76d00d2390 con 0x7f76d00b7bf0 2023-12-13T23:55:21.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f47f80b2f50 con 0x7f47f80cff20 2023-12-13T23:55:21.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f47f4009a20 con 0x7f47f80cff20 2023-12-13T23:55:21.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7f93697fa640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f935006b290 0x7f935006d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbf240b3840 con 0x7fbf24149e50 2023-12-13T23:55:21.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f9371cad640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f934c000fc0 con 0x7f935c005180 2023-12-13T23:55:21.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f936bfff640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f935006b290 0x7f935006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.546+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd234007430 con 0x7fd2440721a0 2023-12-13T23:55:21.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fd22bfff640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd23006b210 0x7fd23006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f936008a200 con 0x7f935c005180 2023-12-13T23:55:21.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47eeffd640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f47dc06b280 0x7f47dc06d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f93697fa640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f93500732c0 0x7f93500756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f47f4089b10 con 0x7f47f80cff20 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9350075d90 con 0x7f93500732c0 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9360054f10 con 0x7f935c005180 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd234057940 con 0x7fd2440721a0 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fd22bfff640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7fd230073430 0x7fd230075850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd230075f00 con 0x7fd230073430 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd234053c80 con 0x7fd2440721a0 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f76c000f3c0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47eeffd640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f47dc073420 0x7f47dc075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47dc075ef0 con 0x7f47dc073420 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47fe8fe640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f47dc073420 0x7f47dc075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47f400be10 con 0x7f47f80cff20 2023-12-13T23:55:21.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f93714ac640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f93500732c0 0x7f93500756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba14009670 con 0x7fba1004fcb0 2023-12-13T23:55:21.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000a700 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.547+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd000a860 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba1400c9a0 con 0x7fba1004fcb0 2023-12-13T23:55:21.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f47fd8fc640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f47dc06b280 0x7f47dc06d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba100e01e0 con 0x7fba1004fcb0 2023-12-13T23:55:21.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000ac50 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbf0c013400 con 0x7fbf24149e50 2023-12-13T23:55:21.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f58671fb640 1 Processor -- start 2023-12-13T23:55:21.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f58671fb640 1 -- start start 2023-12-13T23:55:21.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba100d7790 con 0x7fba1004fcb0 2023-12-13T23:55:21.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f936bfff640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f935006b290 0x7f935006d750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f936c0ad0d0 tx=0x7f9358009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd81f7dd0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd81f75d0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f93714ac640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f93500732c0 0x7f93500756e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba1400cd50 con 0x7fba1004fcb0 2023-12-13T23:55:21.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fbf1bfff640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf0406b2c0 0x7fbf0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fd2437fe640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd23006b210 0x7fd23006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fd243fff640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7fd230073430 0x7fd230075850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f47fe8fe640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f47dc073420 0x7f47dc075840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76b7fff640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f76bc06b290 0x7f76bc06d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f76c0089970 con 0x7f76d00b7bf0 2023-12-13T23:55:21.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76b7fff640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7f76bc073430 0x7f76bc075850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76bc075f00 con 0x7f76bc073430 2023-12-13T23:55:21.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76c00547c0 con 0x7f76d00b7bf0 2023-12-13T23:55:21.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf0c089ae0 con 0x7fbf24149e50 2023-12-13T23:55:21.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7fbf1bfff640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7fbf040734e0 0x7fbf04075900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbf04075fb0 con 0x7fbf040734e0 2023-12-13T23:55:21.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f58580e2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf0c08e2a0 con 0x7fbf24149e50 2023-12-13T23:55:21.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 0x7f5858009d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580d6ea0 0x7f58580d6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580e2970 con 0x7f584c003970 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58580dfc20 con 0x7f58580d7540 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58659f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 0x7f5858009d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58659f8640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 0x7f5858009d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51516/0 (socket says 172.21.15.124:51516) 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58659f8640 1 -- 172.21.15.124:0/3766070118 learned_addr learned my addr 172.21.15.124:0/3766070118 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58669fa640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580d6ea0 0x7f58580d6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb9f8000fc0 con 0x7fba1004fcb0 2023-12-13T23:55:21.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58671fb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58580df2f0 con 0x7f58580d6ea0 2023-12-13T23:55:21.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.548+0000 7f76ceffd640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f76bc06b290 0x7f76bc06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f76cffff640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7f76bc073430 0x7f76bc075850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.549+0000 7f58661f9640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f58580e2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7fbf1a7fc640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf0406b2c0 0x7fbf0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== osd.1 v2:172.21.15.124:6800/3276151417 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9350075d90 con 0x7f93500732c0 2023-12-13T23:55:21.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7fbf237fe640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7fbf040734e0 0x7fbf04075900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1cd815d5c0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 -- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580d6ea0 msgr2=0x7f58580d6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f58580d6ea0 0x7f58580d6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f76cffff640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7f76bc073430 0x7f76bc075850 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 -- 172.21.15.124:0/3766070118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 msgr2=0x7f58580e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f584c003970 0x7f58580e2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58580df5d0 con 0x7f58580d7540 2023-12-13T23:55:21.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7fd243fff640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7fd230073430 0x7fd230075850 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f47fd8fc640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f47dc06b280 0x7f47dc06d740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f47f0035000 tx=0x7f47f0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1cd00070d0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f58659f8640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 0x7f5858009d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f585c00c7f0 tx=0x7f585c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7fbf237fe640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7fbf040734e0 0x7fbf04075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7fbf1a7fc640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf0406b2c0 0x7fbf0406d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbf240b1090 tx=0x7fbf14008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.550+0000 7f76ceffd640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f76bc06b290 0x7f76bc06d750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76d00a1170 tx=0x7f76c4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cbffff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cc406b290 0x7f1cc406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c00cea0 con 0x7f58580d7540 2023-12-13T23:55:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cd00893f0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f585c007500 con 0x7f58580d7540 2023-12-13T23:55:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f585c004a20 con 0x7f58580d7540 2023-12-13T23:55:21.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cbffff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f1cc40732c0 0x7f1cc40756e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1cc4075d90 con 0x7f1cc40732c0 2023-12-13T23:55:21.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== osd.7 v2:172.21.15.203:6800/2404558368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f47dc075ef0 con 0x7f47dc073420 2023-12-13T23:55:21.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1cd00019a0 con 0x7f1cd80d1fa0 2023-12-13T23:55:21.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58580df8b0 con 0x7f58580d7540 2023-12-13T23:55:21.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== osd.0 v2:172.21.15.124:6816/2614715067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd230075f00 con 0x7fd230073430 2023-12-13T23:55:21.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585803cb80 con 0x7f58580d7540 2023-12-13T23:55:21.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fba14013470 con 0x7fba1004fcb0 2023-12-13T23:55:21.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7fd2437fe640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd23006b210 0x7fd23006d6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd238009aa0 tx=0x7fd238009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cd7fff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f1cc40732c0 0x7f1cc40756e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cc406b290 0x7f1cc406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.551+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== osd.2 v2:172.21.15.124:6808/1670235117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbf04075fb0 con 0x7fbf040734e0 2023-12-13T23:55:21.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba0dffb640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9fc06b290 0x7fb9fc06d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba14089b50 con 0x7fba1004fcb0 2023-12-13T23:55:21.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba0dffb640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fb9fc0734b0 0x7fb9fc0758d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== osd.6 v2:172.21.15.203:6808/3358010181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f76bc075f00 con 0x7f76bc073430 2023-12-13T23:55:21.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb9fc075f80 con 0x7fb9fc0734b0 2023-12-13T23:55:21.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba1408e2a0 con 0x7fba1004fcb0 2023-12-13T23:55:21.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7f1cd7fff640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f1cc40732c0 0x7f1cc40756e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9fc06b290 0x7fb9fc06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.552+0000 7fba1db4d640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fb9fc0734b0 0x7fb9fc0758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.553+0000 7f5854ff9640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f585803bf90 con 0x7f58580d7540 2023-12-13T23:55:21.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.553+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== osd.3 v2:172.21.15.124:6817/3893379194 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f1cc4075d90 con 0x7f1cc40732c0 2023-12-13T23:55:21.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.553+0000 7fba1db4d640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fb9fc0734b0 0x7fb9fc0758d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.553+0000 7f1cd6ffd640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cc406b290 0x7f1cc406d750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8034000 tx=0x7f1cc8008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.554+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== osd.4 v2:172.21.15.203:6809/3737544324 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb9fc075f80 con 0x7fb9fc0734b0 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.554+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f585c004b80 con 0x7f58580d7540 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.555+0000 7f5856ffd640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f584406b290 0x7f584406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.555+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f585c089950 con 0x7f58580d7540 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.555+0000 7f5856ffd640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f58440734b0 0x7f58440758d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.555+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5844075f80 con 0x7f58440734b0 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.555+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f585c089d00 con 0x7f58580d7540 2023-12-13T23:55:21.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.556+0000 7fba1cb4b640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9fc06b290 0x7fb9fc06d750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba1000dad0 tx=0x7fba1806d030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.556+0000 7f58661f9640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f584406b290 0x7f584406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.556+0000 7f58669fa640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f58440734b0 0x7f58440758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.556+0000 7f58661f9640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f584406b290 0x7f584406d750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f584c005570 tx=0x7f58600aa000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.557+0000 7f58669fa640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f58440734b0 0x7f58440758d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.557+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== osd.5 v2:172.21.15.203:6824/1613580500 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5844075f80 con 0x7f58440734b0 2023-12-13T23:55:21.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.582+0000 7f9371cad640 1 -- 172.21.15.124:0/4195712959 --> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f934c002c80 con 0x7f93500732c0 2023-12-13T23:55:21.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.583+0000 7f93697fa640 1 -- 172.21.15.124:0/4195712959 <== osd.1 v2:172.21.15.124:6800/3276151417 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f934c002c80 con 0x7f93500732c0 2023-12-13T23:55:21.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.583+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f93500732c0 msgr2=0x7f93500756e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.583+0000 7f9356ffd640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f93500732c0 0x7f93500756e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.583+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f935006b290 msgr2=0x7f935006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f935006b290 0x7f935006d750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f936c0ad0d0 tx=0x7f9358009040 comp rx=0 tx=0).stop 2023-12-13T23:55:21.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 msgr2=0x7f936c06a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7fd248878640 1 -- 172.21.15.124:0/2279747603 --> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd24414db80 con 0x7fd230073430 2023-12-13T23:55:21.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 --2- 172.21.15.124:0/4195712959 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f935c005180 0x7f936c06a740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f93600115d0 tx=0x7f9360011a90 comp rx=0 tx=0).stop 2023-12-13T23:55:21.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f93714ac640 1 -- 172.21.15.124:0/4195712959 reap_dead start 2023-12-13T23:55:21.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 shutdown_connections 2023-12-13T23:55:21.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 >> 172.21.15.124:0/4195712959 conn(0x7f936c0b1a60 msgr2=0x7f936c0b3d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 --> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47f8069320 con 0x7f47dc073420 2023-12-13T23:55:21.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7fd22bfff640 1 -- 172.21.15.124:0/2279747603 <== osd.0 v2:172.21.15.124:6816/2614715067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd24414db80 con 0x7fd230073430 2023-12-13T23:55:21.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 shutdown_connections 2023-12-13T23:55:21.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7f9356ffd640 1 -- 172.21.15.124:0/4195712959 wait complete. 2023-12-13T23:55:21.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.584+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 --> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbf240b3560 con 0x7fbf040734e0 2023-12-13T23:55:21.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47eeffd640 1 -- 172.21.15.124:0/3868715985 <== osd.7 v2:172.21.15.203:6800/2404558368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f47f8069320 con 0x7f47dc073420 2023-12-13T23:55:21.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7fd230073430 msgr2=0x7fd230075850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7fd230073430 0x7fd230075850 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fbf1bfff640 1 -- 172.21.15.124:0/3881174112 <== osd.2 v2:172.21.15.124:6808/1670235117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbf1c001eb0 con 0x7fbf040734e0 2023-12-13T23:55:21.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 --> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1cd81f7760 con 0x7f1cc40732c0 2023-12-13T23:55:21.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd23006b210 msgr2=0x7fd23006d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd23006b210 0x7fd23006d6d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd238009aa0 tx=0x7fd238009190 comp rx=0 tx=0).stop 2023-12-13T23:55:21.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 --> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76d0164b90 con 0x7f76bc073430 2023-12-13T23:55:21.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f47dc073420 msgr2=0x7f47dc075840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f47dc073420 0x7f47dc075840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 msgr2=0x7fd24414f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 --2- 172.21.15.124:0/2279747603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2440721a0 0x7fd24414f650 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd23400c390 tx=0x7fd23400c850 comp rx=0 tx=0).stop 2023-12-13T23:55:21.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd243fff640 1 -- 172.21.15.124:0/2279747603 reap_dead start 2023-12-13T23:55:21.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 shutdown_connections 2023-12-13T23:55:21.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 >> 172.21.15.124:0/2279747603 conn(0x7fd2440b1a60 msgr2=0x7fd2440b3870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f47dc06b280 msgr2=0x7f47dc06d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f47dc06b280 0x7f47dc06d740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f47f0035000 tx=0x7f47f0030000 comp rx=0 tx=0).stop 2023-12-13T23:55:21.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7fbf040734e0 msgr2=0x7fbf04075900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7fbf29533640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7fbf040734e0 0x7fbf04075900 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf0406b2c0 msgr2=0x7fbf0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf0406b2c0 0x7fbf0406d780 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbf240b1090 tx=0x7fbf14008040 comp rx=0 tx=0).stop 2023-12-13T23:55:21.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cbffff640 1 -- 172.21.15.124:0/3704653029 <== osd.3 v2:172.21.15.124:6817/3893379194 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1cd81f7760 con 0x7f1cc40732c0 2023-12-13T23:55:21.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f76b7fff640 1 -- 172.21.15.124:0/812908571 <== osd.6 v2:172.21.15.203:6808/3358010181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f76d0164b90 con 0x7f76bc073430 2023-12-13T23:55:21.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.585+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 msgr2=0x7f47f814f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f47ff0ff640 1 --2- 172.21.15.124:0/3868715985 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f47f80cff20 0x7f47f814f200 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f47f400d830 tx=0x7f47f400dcf0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 msgr2=0x7fbf24160010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 --2- 172.21.15.124:0/3881174112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf24149e50 0x7fbf24160010 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c011510 tx=0x7fbf0c0119d0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf237fe640 1 -- 172.21.15.124:0/3881174112 reap_dead start 2023-12-13T23:55:21.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 shutdown_connections 2023-12-13T23:55:21.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 >> 172.21.15.124:0/3881174112 conn(0x7fbf240cf410 msgr2=0x7fbf24159e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 shutdown_connections 2023-12-13T23:55:21.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fd229ffb640 1 -- 172.21.15.124:0/2279747603 wait complete. 2023-12-13T23:55:21.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f47fe8fe640 1 -- 172.21.15.124:0/3868715985 reap_dead start 2023-12-13T23:55:21.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 shutdown_connections 2023-12-13T23:55:21.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 >> 172.21.15.124:0/3868715985 conn(0x7f47f80d2fd0 msgr2=0x7f47f815d530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 shutdown_connections 2023-12-13T23:55:21.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f1cc40732c0 msgr2=0x7f1cc40756e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f1cc40732c0 0x7f1cc40756e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7fbf29533640 1 -- 172.21.15.124:0/3881174112 wait complete. 2023-12-13T23:55:21.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cc406b290 msgr2=0x7f1cc406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cc406b290 0x7f1cc406d750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8034000 tx=0x7f1cc8008030 comp rx=0 tx=0).stop 2023-12-13T23:55:21.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 msgr2=0x7f1cd815c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 --2- 172.21.15.124:0/3704653029 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd80d1fa0 0x7f1cd815c460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000caf0 tx=0x7f1cd0002080 comp rx=0 tx=0).stop 2023-12-13T23:55:21.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cd7fff640 1 -- 172.21.15.124:0/3704653029 reap_dead start 2023-12-13T23:55:21.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 shutdown_connections 2023-12-13T23:55:21.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.586+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 >> 172.21.15.124:0/3704653029 conn(0x7f1cd809c300 msgr2=0x7f1cd80ce760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7f76bc073430 msgr2=0x7f76bc075850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7f76bc073430 0x7f76bc075850 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 shutdown_connections 2023-12-13T23:55:21.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f1cddfdc640 1 -- 172.21.15.124:0/3704653029 wait complete. 2023-12-13T23:55:21.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f76bc06b290 msgr2=0x7f76bc06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f76bc06b290 0x7f76bc06d750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76d00a1170 tx=0x7f76c4009040 comp rx=0 tx=0).stop 2023-12-13T23:55:21.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 msgr2=0x7f76d0153f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 shutdown_connections 2023-12-13T23:55:21.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f47ff0ff640 1 -- 172.21.15.124:0/3868715985 wait complete. 2023-12-13T23:55:21.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 --2- 172.21.15.124:0/812908571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f76d00b7bf0 0x7f76d0153f20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f76c000b480 tx=0x7f76c000b940 comp rx=0 tx=0).stop 2023-12-13T23:55:21.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 --> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb9f8002cb0 con 0x7fb9fc0734b0 2023-12-13T23:55:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76cffff640 1 -- 172.21.15.124:0/812908571 reap_dead start 2023-12-13T23:55:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 shutdown_connections 2023-12-13T23:55:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 >> 172.21.15.124:0/812908571 conn(0x7f76d00d31f0 msgr2=0x7f76d00b5320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 shutdown_connections 2023-12-13T23:55:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.587+0000 7f76d5f1d640 1 -- 172.21.15.124:0/812908571 wait complete. 2023-12-13T23:55:21.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.588+0000 7fba0dffb640 1 -- 172.21.15.124:0/637841442 <== osd.4 v2:172.21.15.203:6809/3737544324 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb9f8002cb0 con 0x7fb9fc0734b0 2023-12-13T23:55:21.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.588+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fb9fc0734b0 msgr2=0x7fb9fc0758d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.588+0000 7fba1e34e640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fb9fc0734b0 0x7fb9fc0758d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9fc06b290 msgr2=0x7fb9fc06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9fc06b290 0x7fb9fc06d750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba1000dad0 tx=0x7fba1806d030 comp rx=0 tx=0).stop 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 msgr2=0x7fba100d7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 --2- 172.21.15.124:0/637841442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba1004fcb0 0x7fba100d7250 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fba1400e9b0 tx=0x7fba1400ee70 comp rx=0 tx=0).stop 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1db4d640 1 -- 172.21.15.124:0/637841442 reap_dead start 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 shutdown_connections 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 >> 172.21.15.124:0/637841442 conn(0x7fba10054ee0 msgr2=0x7fba10038520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 shutdown_connections 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7fba1e34e640 1 -- 172.21.15.124:0/637841442 wait complete. 2023-12-13T23:55:21.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.589+0000 7f5854ff9640 1 -- 172.21.15.124:0/3766070118 --> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5858006380 con 0x7f58440734b0 2023-12-13T23:55:21.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.591+0000 7f5856ffd640 1 -- 172.21.15.124:0/3766070118 <== osd.5 v2:172.21.15.203:6824/1613580500 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5850004040 con 0x7f58440734b0 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.591+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f58440734b0 msgr2=0x7f58440758d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f58440734b0 0x7f58440758d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f584406b290 msgr2=0x7f584406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f584406b290 0x7f584406d750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f584c005570 tx=0x7f58600aa000 comp rx=0 tx=0).stop 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 msgr2=0x7f5858009d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 --2- 172.21.15.124:0/3766070118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58580d7540 0x7f5858009d20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f585c00c7f0 tx=0x7f585c00ccb0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58669fa640 1 -- 172.21.15.124:0/3766070118 reap_dead start 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 shutdown_connections 2023-12-13T23:55:21.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 >> 172.21.15.124:0/3766070118 conn(0x7f58580559b0 msgr2=0x7f5858006760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 shutdown_connections 2023-12-13T23:55:21.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.592+0000 7f58671fb640 1 -- 172.21.15.124:0/3766070118 wait complete. 2023-12-13T23:55:21.673 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:21.673 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-13T23:55:21.674 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:21.675 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-13T23:55:21.675 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:21.676 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:21.676 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-13T23:55:21.677 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-13T23:55:21.678 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:21.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-13T23:55:21.679 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:21.680 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-13T23:55:21.681 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:21.681 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-13T23:55:21.682 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:21.682 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-13T23:55:21.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.885+0000 7f6812e17640 1 Processor -- start 2023-12-13T23:55:21.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.884+0000 7fea420bb640 1 Processor -- start 2023-12-13T23:55:21.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.885+0000 7fea420bb640 1 -- start start 2023-12-13T23:55:21.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.885+0000 7f6812e17640 1 -- start start 2023-12-13T23:55:21.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fa9428ec640 1 Processor -- start 2023-12-13T23:55:21.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea420bb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fea3c0cd6c0 0x7fea3c0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea420bb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea3c0d19a0 0x7fea3c0cea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea420bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3c15fbd0 con 0x7fea3c15f6d0 2023-12-13T23:55:21.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea420bb640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3c0cba90 con 0x7fea3c0d19a0 2023-12-13T23:55:21.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea420bb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3c0cbca0 con 0x7fea3c0cd6c0 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f6812e17640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f68040ead00 0x7f680404fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f68113b4640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6804006ea0 0x7f68040382e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43590/0 (socket says 172.21.15.124:43590) 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f68113b4640 1 -- 172.21.15.124:0/3826737424 learned_addr learned my addr 172.21.15.124:0/3826737424 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f6812e17640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804054330 con 0x7f6804006ea0 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6804053720 con 0x7f680404ffd0 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6804053a00 con 0x7f68040ead00 2023-12-13T23:55:21.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea3bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea3c0d19a0 0x7fea3c0cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea3bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea3c0d19a0 0x7fea3c0cea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57200/0 (socket says 172.21.15.124:57200) 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fea3bfff640 1 -- 172.21.15.124:0/95663198 learned_addr learned my addr 172.21.15.124:0/95663198 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fa5a76ec640 1 Processor -- start 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fea3b7fe640 1 --2- 172.21.15.124:0/95663198 >> v2:172.21.15.203:3300/0 conn(0x7fea3c0cd6c0 0x7fea3c0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fea3bfff640 1 -- 172.21.15.124:0/95663198 >> v2:172.21.15.203:3300/0 conn(0x7fea3c0cd6c0 msgr2=0x7fea3c0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fea3bfff640 1 --2- 172.21.15.124:0/95663198 >> v2:172.21.15.203:3300/0 conn(0x7fea3c0cd6c0 0x7fea3c0cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fea3bfff640 1 -- 172.21.15.124:0/95663198 >> v1:172.21.15.203:6789/0 conn(0x7fea3c15f6d0 legacy=0x7fea3c0cdaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.886+0000 7fa9428ec640 1 -- start start 2023-12-13T23:55:21.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fa5a76ec640 1 -- start start 2023-12-13T23:55:21.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fb706bbe640 1 Processor -- start 2023-12-13T23:55:21.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318932819 0 0) 0x7f6804054330 con 0x7f6804006ea0 2023-12-13T23:55:21.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc003520 con 0x7f6804006ea0 2023-12-13T23:55:21.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fa9428ec640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa93c0b0cf0 0x7fa93c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa9428ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c0b2060 0x7fa93c0b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fea3bfff640 1 -- 172.21.15.124:0/95663198 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0cbf80 con 0x7fea3c0d19a0 2023-12-13T23:55:21.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa9428ec640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c162d80 con 0x7fa93c158690 2023-12-13T23:55:21.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3404237348 0 0) 0x7f67fc003520 con 0x7f6804006ea0 2023-12-13T23:55:21.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f6810bb3640 1 --2- 172.21.15.124:0/3826737424 >> v2:172.21.15.203:3300/0 conn(0x7f68040ead00 0x7f680404fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 >> v2:172.21.15.203:3300/0 conn(0x7f68040ead00 msgr2=0x7f680404fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa9428ec640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa93c0b4c00 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 --2- 172.21.15.124:0/3826737424 >> v2:172.21.15.203:3300/0 conn(0x7f68040ead00 0x7f680404fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 >> v1:172.21.15.203:6789/0 conn(0x7f680404ffd0 legacy=0x7f6804037bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6804053ce0 con 0x7f6804006ea0 2023-12-13T23:55:21.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa9428ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c0b16c0 con 0x7fa93c0b2060 2023-12-13T23:55:21.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93bfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa93c158690 0x7fa93c0b44f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43594/0 (socket says 172.21.15.124:43594) 2023-12-13T23:55:21.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93bfff640 1 -- 172.21.15.124:0/3360540290 learned_addr learned my addr 172.21.15.124:0/3360540290 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93b7fe640 1 --2- 172.21.15.124:0/3360540290 >> v2:172.21.15.124:3300/0 conn(0x7fa93c0b0cf0 0x7fa93c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a76ec640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa5a00b1880 0x7fa5a015a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a76ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b47c0 0x7fa5a00b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f680c0689d0 con 0x7f6804006ea0 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 >> v1:172.21.15.124:6789/0 conn(0x7f6804006ea0 legacy=0x7f68040382e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc0039f0 0x7f67fc003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67fc0047c0 0x7f67fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 0x7f67fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804053a00 con 0x7f67fc0047c0 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc00c0b0 con 0x7f67fc005180 2023-12-13T23:55:21.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc00c270 con 0x7f67fc0039f0 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a76ec640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a015a830 con 0x7fa5a0158fa0 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a76ec640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a00cd300 con 0x7fa5a00b1880 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a76ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00cd5e0 con 0x7fa5a00b47c0 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a4c87640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa5a0158fa0 0x7fa5a00b40b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43606/0 (socket says 172.21.15.124:43606) 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a4c87640 1 -- 172.21.15.124:0/708134303 learned_addr learned my addr 172.21.15.124:0/708134303 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a5488640 1 --2- 172.21.15.124:0/708134303 >> v2:172.21.15.203:3300/0 conn(0x7fa5a00b1880 0x7fa5a015a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa5a5c89640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b47c0 0x7fa5a00b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f055359e640 1 Processor -- start 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93b7fe640 1 -- 172.21.15.124:0/3360540290 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c0b2060 msgr2=0x7fa93c0b24d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93b7fe640 1 --2- 172.21.15.124:0/3360540290 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c0b2060 0x7fa93c0b24d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93affd640 1 -- 172.21.15.124:0/3360540290 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551855095 0 0) 0x7fa93c162d80 con 0x7fa93c158690 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93b7fe640 1 -- 172.21.15.124:0/3360540290 >> v1:172.21.15.124:6789/0 conn(0x7fa93c158690 legacy=0x7fa93c0b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7fa93b7fe640 1 -- 172.21.15.124:0/3360540290 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c0705a0 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f68113b4640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 0x7f67fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f6810bb3640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc0039f0 0x7f67fc003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.888+0000 7f680bfff640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67fc0047c0 0x7f67fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fc8e3797640 1 Processor -- start 2023-12-13T23:55:21.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.887+0000 7fb706bbe640 1 -- start start 2023-12-13T23:55:21.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb706bbe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb706bbe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb6f80067e0 0x7fb6f8006c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb706bbe640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f8054300 con 0x7fb6f80eacd0 2023-12-13T23:55:21.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb706bbe640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f80536f0 con 0x7fb6f80067e0 2023-12-13T23:55:21.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb706bbe640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80539d0 con 0x7fb6f8005e60 2023-12-13T23:55:21.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fa5a5488640 1 -- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b47c0 msgr2=0x7fa5a00b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fa5a5488640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b47c0 0x7fa5a00b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fa5a5488640 1 -- 172.21.15.124:0/708134303 >> v1:172.21.15.124:6789/0 conn(0x7fa5a0158fa0 legacy=0x7fa5a00b40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fa5a5488640 1 -- 172.21.15.124:0/708134303 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a00cd8c0 con 0x7fa5a00b1880 2023-12-13T23:55:21.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb70515b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb6f80067e0 0x7fb6f8006c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60056/0 (socket says 172.21.15.124:60056) 2023-12-13T23:55:21.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 -- 172.21.15.124:0/1210981406 learned_addr learned my addr 172.21.15.124:0/1210981406 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fc8e3797640 1 -- start start 2023-12-13T23:55:21.902 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fa93b7fe640 1 --2- 172.21.15.124:0/3360540290 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c0b1180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9300097a0 tx=0x7fa93002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7416d461bd4949a7 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.902 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 -- 172.21.15.124:0/1210981406 >> v2:172.21.15.124:3300/0 conn(0x7fb6f80067e0 msgr2=0x7fb6f8006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f68113b4640 1 -- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc0039f0 msgr2=0x7f67fc003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fa93affd640 1 -- 172.21.15.124:0/3360540290 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930031830 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f68113b4640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc0039f0 0x7f67fc003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f68113b4640 1 -- 172.21.15.124:0/3826737424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67fc0047c0 msgr2=0x7f67fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f68113b4640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67fc0047c0 0x7f67fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f68113b4640 1 -- 172.21.15.124:0/3826737424 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc003520 con 0x7f67fc005180 2023-12-13T23:55:21.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f680bfff640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67fc0047c0 0x7f67fc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:21.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7f68113b4640 1 -- 172.21.15.124:0/3826737424 reap_dead start 2023-12-13T23:55:21.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7f68113b4640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 0x7f67fc00b980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f680c068f20 tx=0x7f67fc0057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31b83271a10f5f9b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fa93affd640 1 -- 172.21.15.124:0/3360540290 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa930031990 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fa5a5488640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b1880 0x7fa5a015a2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0097a0 tx=0x7fa59c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80780af810526302 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc8dc158f90 0x7fc8dc0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc06c4c0 0x7fc8dc15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc8dc071de0 0x7fc8dc072270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc0727b0 con 0x7fc8dc071de0 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8dc15d110 con 0x7fc8dc158f90 2023-12-13T23:55:21.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e3797640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc15d3b0 con 0x7fc8dc06c4c0 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fa93affd640 1 -- 172.21.15.124:0/3360540290 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa930031830 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0aa6f0 con 0x7f67fc005180 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7f680b7fe640 1 -- 172.21.15.124:0/3826737424 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c0aa850 con 0x7f67fc005180 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 --2- 172.21.15.124:0/1210981406 >> v2:172.21.15.124:3300/0 conn(0x7fb6f80067e0 0x7fb6f8006c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 -- 172.21.15.124:0/1210981406 >> v1:172.21.15.203:6789/0 conn(0x7fb6f80eacd0 legacy=0x7fb6f804fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7fb6fffff640 1 -- 172.21.15.124:0/1210981406 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f8053cb0 con 0x7fb6f8005e60 2023-12-13T23:55:21.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e1d34640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc8dc071de0 0x7fc8dc072270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fa597fff640 1 -- 172.21.15.124:0/708134303 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c030800 con 0x7fa5a00b1880 2023-12-13T23:55:21.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e1d34640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc8dc071de0 0x7fc8dc072270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57250/0 (socket says 172.21.15.124:57250) 2023-12-13T23:55:21.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e1d34640 1 -- 172.21.15.124:0/3898912536 learned_addr learned my addr 172.21.15.124:0/3898912536 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.890+0000 7fc8e1533640 1 --2- 172.21.15.124:0/3898912536 >> v2:172.21.15.203:3300/0 conn(0x7fc8dc158f90 0x7fc8dc0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa597fff640 1 -- 172.21.15.124:0/708134303 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c030960 con 0x7fa5a00b1880 2023-12-13T23:55:21.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e0d32640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc06c4c0 0x7fc8dc15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e1d34640 1 -- 172.21.15.124:0/3898912536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc06c4c0 msgr2=0x7fc8dc15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e1d34640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc06c4c0 0x7fc8dc15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e1d34640 1 -- 172.21.15.124:0/3898912536 >> v2:172.21.15.203:3300/0 conn(0x7fc8dc158f90 msgr2=0x7fc8dc0ce290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e1d34640 1 --2- 172.21.15.124:0/3898912536 >> v2:172.21.15.203:3300/0 conn(0x7fc8dc158f90 0x7fc8dc0ce290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.889+0000 7f055359e640 1 -- start start 2023-12-13T23:55:21.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055359e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f054c057450 0x7f054c036d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fc8e1d34640 1 -- 172.21.15.124:0/3898912536 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc15d690 con 0x7fc8dc071de0 2023-12-13T23:55:21.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055359e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c051390 con 0x7f054c04a360 2023-12-13T23:55:21.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c054590 con 0x7f054c049f80 2023-12-13T23:55:21.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055359e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c007c30 con 0x7f054c057450 2023-12-13T23:55:21.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fb6fffff640 1 --2- 172.21.15.124:0/1210981406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009870 tx=0x7fb6f402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2db933d57d0fee6a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 -- 172.21.15.124:0/3360540290 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 msgr2=0x7fa93c0b1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 --2- 172.21.15.124:0/3360540290 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c0b1180 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa9300097a0 tx=0x7fa93002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 msgr2=0x7f67fc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f6812e17640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 0x7f67fc00b980 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f680c068f20 tx=0x7f67fc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 shutdown_connections 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f6812e17640 1 --2- 172.21.15.124:0/3826737424 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67fc005180 0x7f67fc00b980 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 >> 172.21.15.124:0/3826737424 conn(0x7f6804009fb0 msgr2=0x7f680400a390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fea3bfff640 1 --2- 172.21.15.124:0/95663198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0d19a0 0x7fea3c0cea50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea3c069740 tx=0x7fea3402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1c08fef8b67bca server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fea3a7fc640 1 -- 172.21.15.124:0/95663198 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3402ca80 con 0x7fea3c0d19a0 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fea3a7fc640 1 -- 172.21.15.124:0/95663198 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3402cbe0 con 0x7fea3c0d19a0 2023-12-13T23:55:21.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fb6ff7fe640 1 -- 172.21.15.124:0/1210981406 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4031830 con 0x7fb6f8005e60 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fb6ff7fe640 1 -- 172.21.15.124:0/1210981406 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f4031990 con 0x7fb6f8005e60 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa597fff640 1 -- 172.21.15.124:0/708134303 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c030cf0 con 0x7fa5a00b1880 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 -- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b1880 msgr2=0x7fa5a015a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b1880 0x7fa5a015a2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa59c0097a0 tx=0x7fa59c02b940 comp rx=0 tx=0).stop 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 -- 172.21.15.124:0/708134303 shutdown_connections 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b47c0 0x7fa5a00b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/708134303 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b1880 0x7fa5a015a2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 -- 172.21.15.124:0/708134303 >> 172.21.15.124:0/708134303 conn(0x7fa5a00b0d60 msgr2=0x7fa5a00b1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 -- 172.21.15.124:0/708134303 shutdown_connections 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fa5a76ec640 1 -- 172.21.15.124:0/708134303 wait complete. 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055259c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f054c049f80 0x7f054c051820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33076/0 (socket says 172.21.15.124:33076) 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f055259c640 1 -- 172.21.15.124:0/2648753622 learned_addr learned my addr 172.21.15.124:0/2648753622 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7f0551d9b640 1 --2- 172.21.15.124:0/2648753622 >> v2:172.21.15.203:3300/0 conn(0x7f054c057450 0x7f054c036d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f055159a640 1 -- 172.21.15.124:0/2648753622 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533565341 0 0) 0x7f054c054590 con 0x7f054c049f80 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f055159a640 1 -- 172.21.15.124:0/2648753622 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f053c003520 con 0x7f054c049f80 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f0551d9b640 1 -- 172.21.15.124:0/2648753622 >> v1:172.21.15.203:6789/0 conn(0x7f054c049f80 legacy=0x7f054c051820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f055159a640 1 -- 172.21.15.124:0/2648753622 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1022718924 0 0) 0x7f053c003520 con 0x7f054c049f80 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 -- 172.21.15.124:0/3360540290 shutdown_connections 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 --2- 172.21.15.124:0/3360540290 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c0b2060 0x7fa93c0b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 --2- 172.21.15.124:0/3360540290 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c0b1180 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 -- 172.21.15.124:0/3360540290 >> 172.21.15.124:0/3360540290 conn(0x7fa93c06a4d0 msgr2=0x7fa93c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 -- 172.21.15.124:0/3360540290 shutdown_connections 2023-12-13T23:55:21.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.891+0000 7fa9428ec640 1 -- 172.21.15.124:0/3360540290 wait complete. 2023-12-13T23:55:21.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f0551d9b640 1 -- 172.21.15.124:0/2648753622 >> v1:172.21.15.124:6789/0 conn(0x7f054c04a360 legacy=0x7f054c051f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f0551d9b640 1 -- 172.21.15.124:0/2648753622 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c007f10 con 0x7f054c057450 2023-12-13T23:55:21.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fea3a7fc640 1 -- 172.21.15.124:0/95663198 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea340343f0 con 0x7fea3c0d19a0 2023-12-13T23:55:21.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fb706bbe640 1 -- 172.21.15.124:0/1210981406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 msgr2=0x7fb6f80062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fb706bbe640 1 --2- 172.21.15.124:0/1210981406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009870 tx=0x7fb6f402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7fc8e1d34640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc071de0 0x7fc8dc072270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc071630 tx=0x7fc8d402d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685f4336eaf04634 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 -- 172.21.15.124:0/95663198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0d19a0 msgr2=0x7fea3c0cea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 --2- 172.21.15.124:0/95663198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0d19a0 0x7fea3c0cea50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fea3c069740 tx=0x7fea3402bee0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 -- 172.21.15.124:0/1210981406 shutdown_connections 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 --2- 172.21.15.124:0/1210981406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f80062a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 --2- 172.21.15.124:0/1210981406 >> v2:172.21.15.124:3300/0 conn(0x7fb6f80067e0 0x7fb6f8006c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 -- 172.21.15.124:0/1210981406 >> 172.21.15.124:0/1210981406 conn(0x7fb6f8009f80 msgr2=0x7fb6f800a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 shutdown_connections 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.892+0000 7f6812e17640 1 -- 172.21.15.124:0/3826737424 wait complete. 2023-12-13T23:55:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fc8d3fff640 1 -- 172.21.15.124:0/3898912536 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400a4e0 con 0x7fc8dc071de0 2023-12-13T23:55:21.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 -- 172.21.15.124:0/1210981406 shutdown_connections 2023-12-13T23:55:21.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fc8d3fff640 1 -- 172.21.15.124:0/3898912536 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d4030050 con 0x7fc8dc071de0 2023-12-13T23:55:21.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fc8d3fff640 1 -- 172.21.15.124:0/3898912536 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d400a8a0 con 0x7fc8dc071de0 2023-12-13T23:55:21.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7f0551d9b640 1 --2- 172.21.15.124:0/2648753622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c057450 0x7f054c036d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0548009720 tx=0x7f054802b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7ff1d88dbdec097 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 -- 172.21.15.124:0/95663198 shutdown_connections 2023-12-13T23:55:21.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 --2- 172.21.15.124:0/95663198 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0d19a0 0x7fea3c0cea50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 --2- 172.21.15.124:0/95663198 >> v2:172.21.15.203:3300/0 conn(0x7fea3c0cd6c0 0x7fea3c0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 -- 172.21.15.124:0/95663198 >> 172.21.15.124:0/95663198 conn(0x7fea3c0d0e80 msgr2=0x7fea3c0d1290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 -- 172.21.15.124:0/95663198 shutdown_connections 2023-12-13T23:55:21.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fea420bb640 1 -- 172.21.15.124:0/95663198 wait complete. 2023-12-13T23:55:21.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fa9428ec640 1 Processor -- start 2023-12-13T23:55:21.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.893+0000 7fb706bbe640 1 -- 172.21.15.124:0/1210981406 wait complete. 2023-12-13T23:55:21.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fa5a76ec640 1 Processor -- start 2023-12-13T23:55:21.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fa9428ec640 1 -- start start 2023-12-13T23:55:21.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 -- 172.21.15.124:0/3898912536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc071de0 msgr2=0x7fc8dc072270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc071de0 0x7fc8dc072270 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc071630 tx=0x7fc8d402d3b0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 -- 172.21.15.124:0/3898912536 shutdown_connections 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc071de0 0x7fc8dc072270 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 --2- 172.21.15.124:0/3898912536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc06c4c0 0x7fc8dc15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 --2- 172.21.15.124:0/3898912536 >> v2:172.21.15.203:3300/0 conn(0x7fc8dc158f90 0x7fc8dc0ce290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 -- 172.21.15.124:0/3898912536 >> 172.21.15.124:0/3898912536 conn(0x7fc8dc0b0b10 msgr2=0x7fc8dc0b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 -- 172.21.15.124:0/3898912536 shutdown_connections 2023-12-13T23:55:21.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7f7d6d852640 1 Processor -- start 2023-12-13T23:55:21.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fb706bbe640 1 Processor -- start 2023-12-13T23:55:21.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fb706bbe640 1 -- start start 2023-12-13T23:55:21.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fc8e3797640 1 -- 172.21.15.124:0/3898912536 wait complete. 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7f7d6d852640 1 -- start start 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7fa5a76ec640 1 -- start start 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b47c0 0x7fa5a01e5f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a0158fa0 0x7fa5a01e64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00a84d0 con 0x7fa5a0158fa0 2023-12-13T23:55:21.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a00ab4f0 con 0x7fa5a00b47c0 2023-12-13T23:55:21.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a76ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a01e8d60 con 0x7fa5a00b1880 2023-12-13T23:55:21.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fea420bb640 1 Processor -- start 2023-12-13T23:55:21.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7f6812e17640 1 Processor -- start 2023-12-13T23:55:21.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.894+0000 7f6812e17640 1 -- start start 2023-12-13T23:55:21.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa5a5c89640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a0158fa0 0x7fa5a01e64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa93c0b2060 0x7fa93c161a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c158690 0x7fa93c165150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c1f7690 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc005180 0x7f680404e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68041888e0 0x7f68040358b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804189050 con 0x7f68041888e0 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6804189330 con 0x7f6804185a40 2023-12-13T23:55:21.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7f6812e17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6804189610 con 0x7f67fc005180 2023-12-13T23:55:21.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c162280 con 0x7fa93c0b2060 2023-12-13T23:55:21.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fa9428ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c162560 con 0x7fa93c158690 2023-12-13T23:55:21.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.895+0000 7fea420bb640 1 -- start start 2023-12-13T23:55:21.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57256/0 (socket says 172.21.15.124:57256) 2023-12-13T23:55:21.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 -- 172.21.15.124:0/2973236335 learned_addr learned my addr 172.21.15.124:0/2973236335 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51582/0 (socket says 172.21.15.124:51582) 2023-12-13T23:55:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 -- 172.21.15.124:0/4092358519 learned_addr learned my addr 172.21.15.124:0/4092358519 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f6810bb3640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc005180 0x7f680404e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60074/0 (socket says 172.21.15.124:60074) 2023-12-13T23:55:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 -- 172.21.15.124:0/1282468168 learned_addr learned my addr 172.21.15.124:0/1282468168 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6d852640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 -- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc005180 msgr2=0x7f680404e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc005180 0x7f680404e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 -- 172.21.15.124:0/4092358519 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68041888e0 msgr2=0x7f68040358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68041888e0 0x7f68040358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 -- 172.21.15.124:0/4092358519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6804185520 con 0x7f6804185a40 2023-12-13T23:55:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a4c87640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b47c0 0x7fa5a01e5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fc8e3797640 1 Processor -- start 2023-12-13T23:55:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 -- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c158690 msgr2=0x7fa93c165150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c158690 0x7fa93c165150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 -- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa93c0b2060 msgr2=0x7fa93c161a30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa93c0b2060 0x7fa93c161a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa93bfff640 1 -- 172.21.15.124:0/2973236335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa930009400 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6d852640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7d600e0070 0x7f7d60038db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6d852640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d60007ad0 con 0x7f7d600392f0 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6d852640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d60050900 con 0x7f7d600e0070 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6d852640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d600e6aa0 con 0x7f7d60050400 2023-12-13T23:55:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6c850640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6c850640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60102/0 (socket says 172.21.15.124:60102) 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f7d6c850640 1 -- 172.21.15.124:0/1798316338 learned_addr learned my addr 172.21.15.124:0/1798316338 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 -- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b47c0 msgr2=0x7fa5a01e5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b47c0 0x7fa5a01e5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 -- 172.21.15.124:0/1282468168 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a0158fa0 msgr2=0x7fa5a01e64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fc8e3797640 1 -- start start 2023-12-13T23:55:21.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f055159a640 1 -- 172.21.15.124:0/2648753622 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05540687b0 con 0x7f054c057450 2023-12-13T23:55:21.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a0158fa0 0x7fa5a01e64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fa5a5488640 1 -- 172.21.15.124:0/1282468168 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c009400 con 0x7fa5a00b1880 2023-12-13T23:55:21.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f055159a640 1 -- 172.21.15.124:0/2648753622 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054802e050 con 0x7f054c057450 2023-12-13T23:55:21.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7f68113b4640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f680c04f5f0 tx=0x7f680c0743c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.896+0000 7fb706bbe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f8049940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fb706bbe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f7d67fff640 1 --2- 172.21.15.124:0/1798316338 >> v2:172.21.15.203:3300/0 conn(0x7f7d600e0070 0x7f7d60038db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f7d6c850640 1 -- 172.21.15.124:0/1798316338 >> v2:172.21.15.203:3300/0 conn(0x7f7d600e0070 msgr2=0x7f7d60038db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:21.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fb70495a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f8049940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f7d6c850640 1 --2- 172.21.15.124:0/1798316338 >> v2:172.21.15.203:3300/0 conn(0x7f7d600e0070 0x7f7d60038db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa93bfff640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c430 tx=0x7fa92c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f7d6c850640 1 -- 172.21.15.124:0/1798316338 >> v1:172.21.15.124:6789/0 conn(0x7f7d600392f0 legacy=0x7f7d60008ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f7d6c850640 1 -- 172.21.15.124:0/1798316338 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d60034ce0 con 0x7f7d60050400 2023-12-13T23:55:21.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0b6040 con 0x7f6804185a40 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c04ed60 con 0x7f6804185a40 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f680c0aad40 con 0x7f6804185a40 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6804188d60 con 0x7f6804185a40 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6804035fd0 con 0x7f6804185a40 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0cd6c0 0x7fea3c149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 0x7fea3c0c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 0x7fea3c0aec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0c5740 con 0x7fea3c0cd6c0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea3affd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 0x7fea3c0c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea3affd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 0x7fea3c0c93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60124/0 (socket says 172.21.15.124:60124) 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea3affd640 1 -- 172.21.15.124:0/1987642608 learned_addr learned my addr 172.21.15.124:0/1987642608 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea3bfff640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 0x7fea3c0aec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0c4b30 con 0x7fea3c15f6d0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3c0c4e10 con 0x7fea3c0d19a0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fb6fffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fb6fffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57286/0 (socket says 172.21.15.124:57286) 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fb6fffff640 1 -- 172.21.15.124:0/303151247 learned_addr learned my addr 172.21.15.124:0/303151247 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f80eacd0 0x7fb6f80e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f817d0d0 con 0x7fb6f80067e0 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80e5880 con 0x7fb6f80eacd0 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 0x7fc8dc157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 0x7fc8dc13b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f055359e640 1 -- 172.21.15.124:0/2648753622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c057450 msgr2=0x7f054c036d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7f055359e640 1 --2- 172.21.15.124:0/2648753622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c057450 0x7f054c036d80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f0548009720 tx=0x7f054802b6e0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c0074e0 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa92c007660 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa92c004a20 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c162840 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f80e5b60 con 0x7fb6f8005e60 2023-12-13T23:55:21.945 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.897+0000 7fa5a5488640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa59c02bf50 tx=0x7fa59c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.945 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc158f90 0x7fc8dc0a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.945 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7f055359e640 1 -- 172.21.15.124:0/2648753622 shutdown_connections 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fea3bfff640 1 -- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 msgr2=0x7fea3c0c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fea3bfff640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 0x7fea3c0c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fea3bfff640 1 -- 172.21.15.124:0/1987642608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0cd6c0 msgr2=0x7fea3c149210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fea3bfff640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0cd6c0 0x7fea3c149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fea3bfff640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea34009b30 con 0x7fea3c15f6d0 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7f055359e640 1 --2- 172.21.15.124:0/2648753622 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c057450 0x7f054c036d80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7f055359e640 1 -- 172.21.15.124:0/2648753622 >> 172.21.15.124:0/2648753622 conn(0x7f054c020010 msgr2=0x7f054c056d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1557c0 con 0x7fc8dc06c4c0 2023-12-13T23:55:21.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0a99d0 con 0x7fc8dc071de0 2023-12-13T23:55:21.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e3797640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc0a96f0 con 0x7fc8dc158f90 2023-12-13T23:55:21.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fb70515b640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f80eacd0 0x7fb6f80e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e1533640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 0x7fc8dc157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e1533640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 0x7fc8dc157830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57304/0 (socket says 172.21.15.124:57304) 2023-12-13T23:55:21.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e1533640 1 -- 172.21.15.124:0/1519813092 learned_addr learned my addr 172.21.15.124:0/1519813092 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c161f70 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7f7d6c850640 1 --2- 172.21.15.124:0/1798316338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02b9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d44d00b8c5a5516 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fc8e0d32640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 0x7fc8dc13b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e1d34640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc158f90 0x7fc8dc0a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 -- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 msgr2=0x7fb6f8049940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f8049940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 -- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f80eacd0 msgr2=0x7fb6f80e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f80eacd0 0x7fb6f80e5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f40094d0 con 0x7fb6f80067e0 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c03e040 con 0x7fa5a00b1880 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa59c039c70 con 0x7fa5a00b1880 2023-12-13T23:55:21.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.898+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59c0443f0 con 0x7fa5a00b1880 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f7d677fe640 1 -- 172.21.15.124:0/1798316338 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c030840 con 0x7f7d60050400 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f7d677fe640 1 -- 172.21.15.124:0/1798316338 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c0309a0 con 0x7f7d60050400 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a01e8200 con 0x7fa5a00b1880 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a00a8a00 con 0x7fa5a00b1880 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f680c0b9070 con 0x7f6804185a40 2023-12-13T23:55:21.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6804189830 con 0x7f6804185a40 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f68097fa640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 0x7f67f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f680c125ba0 con 0x7f6804185a40 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fb6fffff640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6f402b910 tx=0x7fb6f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f403acd0 con 0x7fb6f80067e0 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f403ae30 con 0x7fb6f80067e0 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f80e5e40 con 0x7fb6f80067e0 2023-12-13T23:55:21.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f4044630 con 0x7fb6f80067e0 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e0d32640 1 -- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc158f90 msgr2=0x7fc8dc0a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e0d32640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc158f90 0x7fc8dc0a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e0d32640 1 -- 172.21.15.124:0/1519813092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 msgr2=0x7fc8dc157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e0d32640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 0x7fc8dc157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fc8e0d32640 1 -- 172.21.15.124:0/1519813092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d400c050 con 0x7fc8dc071de0 2023-12-13T23:55:21.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7f055359e640 1 -- 172.21.15.124:0/2648753622 shutdown_connections 2023-12-13T23:55:21.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa93c1f7800 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 -- 172.21.15.124:0/1798316338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 msgr2=0x7f7d60034f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d677fe640 1 -- 172.21.15.124:0/1798316338 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c030840 con 0x7f7d60050400 2023-12-13T23:55:21.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 --2- 172.21.15.124:0/1798316338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f80e6400 con 0x7fb6f80067e0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fea3bfff640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 0x7fea3c0aec00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea34032000 tx=0x7fea34004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3403e450 con 0x7fea3c15f6d0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3403e5b0 con 0x7fea3c15f6d0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.899+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3403e960 con 0x7fea3c15f6d0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3c0c50f0 con 0x7fea3c15f6d0 2023-12-13T23:55:21.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fc8e0d32640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 0x7fc8dc13b580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800e870 tx=0x7fc8d800ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa92c004b80 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 -- 172.21.15.124:0/1798316338 shutdown_connections 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 --2- 172.21.15.124:0/1798316338 >> v2:172.21.15.203:3300/0 conn(0x7f7d600e0070 0x7f7d60038db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 --2- 172.21.15.124:0/1798316338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d60050400 0x7f7d60034f90 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 -- 172.21.15.124:0/1798316338 >> 172.21.15.124:0/1798316338 conn(0x7f7d60055ba0 msgr2=0x7f7d60055fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:21.956 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 -- 172.21.15.124:0/1798316338 shutdown_connections 2023-12-13T23:55:21.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3c15a030 con 0x7fea3c15f6d0 2023-12-13T23:55:21.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f7d6d852640 1 -- 172.21.15.124:0/1798316338 wait complete. 2023-12-13T23:55:21.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f6810bb3640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 0x7f67f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fa938ff9640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 0x7fa92806d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a0069400 con 0x7fa5a00b1880 2023-12-13T23:55:21.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.900+0000 7f055359e640 1 -- 172.21.15.124:0/2648753622 wait complete. 2023-12-13T23:55:21.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fa93b7fe640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 0x7fa92806d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.901+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa59c04d3b0 con 0x7fa5a00b1880 2023-12-13T23:55:21.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa92c0897e0 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7f6810bb3640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 0x7f67f006d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6804003e80 tx=0x7f6800008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fa595ffb640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 0x7fa58406d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa59c0abf50 con 0x7fa5a00b1880 2023-12-13T23:55:21.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7f055359e640 1 Processor -- start 2023-12-13T23:55:21.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7f055359e640 1 -- start start 2023-12-13T23:55:21.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb6f404d450 con 0x7fb6f80067e0 2023-12-13T23:55:21.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fea21ffb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3c066d00 con 0x7fea3c15f6d0 2023-12-13T23:55:21.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7fa5a4c87640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 0x7fa58406d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.902+0000 7f7d6d852640 1 Processor -- start 2023-12-13T23:55:21.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7fa93b7fe640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 0x7fa92806d6d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa930036000 tx=0x7fa930008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f7d6d852640 1 -- start start 2023-12-13T23:55:21.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7fa5a4c87640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 0x7fa58406d750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5a01e8fc0 tx=0x7fa598009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f054c04a360 0x7f054c0356d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f054c057450 0x7f054c035c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c036490 con 0x7f054c04a360 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c036770 con 0x7f054c049f80 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055359e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0d6ca0 con 0x7f054c057450 2023-12-13T23:55:21.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055259c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51628/0 (socket says 172.21.15.124:51628) 2023-12-13T23:55:21.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f055259c640 1 -- 172.21.15.124:0/3077213767 learned_addr learned my addr 172.21.15.124:0/3077213767 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f0551d9b640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f054c04a360 0x7f054c0356d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7f0552d9d640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f054c057450 0x7f054c035c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6f8005360 con 0x7fb6f80067e0 2023-12-13T23:55:21.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.903+0000 7fb6fd7fa640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 0x7fb6e406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 -- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f054c057450 msgr2=0x7f054c035c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb6f40abbb0 con 0x7fb6f80067e0 2023-12-13T23:55:21.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f054c057450 0x7f054c035c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7fb70495a640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 0x7fb6e406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 -- 172.21.15.124:0/3077213767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f054c04a360 msgr2=0x7f054c0356d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f054c04a360 0x7f054c0356d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 -- 172.21.15.124:0/3077213767 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0548009400 con 0x7f054c049f80 2023-12-13T23:55:21.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f055259c640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0554079e00 tx=0x7f0554067f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7fb70495a640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 0x7fb6e406d680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000f60 tx=0x7fb6ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d60050400 0x7f7d600d44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d600e0070 0x7f7d60053fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d600d4a10 con 0x7f7d600392f0 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d600e18c0 con 0x7f7d60050400 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d852640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6004d100 con 0x7f7d600e0070 2023-12-13T23:55:21.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6c850640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6c850640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57312/0 (socket says 172.21.15.124:57312) 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6c850640 1 -- 172.21.15.124:0/2280104460 learned_addr learned my addr 172.21.15.124:0/2280104460 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d67fff640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d60050400 0x7f7d600d44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.904+0000 7f7d6d051640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d600e0070 0x7f7d60053fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 -- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d600e0070 msgr2=0x7f7d60053fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d600e0070 0x7f7d60053fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 -- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d60050400 msgr2=0x7f7d600d44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d60050400 0x7f7d600d44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 -- 172.21.15.124:0/2280104460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5c0094d0 con 0x7f7d600392f0 2023-12-13T23:55:21.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f7d6c850640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000c00 tx=0x7f7d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f055406bdc0 con 0x7f054c049f80 2023-12-13T23:55:21.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d800ee60 con 0x7fc8dc071de0 2023-12-13T23:55:21.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0d6f80 con 0x7f054c049f80 2023-12-13T23:55:21.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc155170 con 0x7fc8dc071de0 2023-12-13T23:55:21.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.905+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c179560 con 0x7f054c049f80 2023-12-13T23:55:21.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.906+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc13bac0 con 0x7fc8dc071de0 2023-12-13T23:55:21.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.906+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05540678c0 con 0x7f054c049f80 2023-12-13T23:55:21.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.906+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f680c0f08b0 con 0x7f6804185a40 2023-12-13T23:55:21.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.906+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05540723e0 con 0x7f054c049f80 2023-12-13T23:55:21.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.906+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d8005990 con 0x7fc8dc071de0 2023-12-13T23:55:21.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc155970 con 0x7fc8dc071de0 2023-12-13T23:55:21.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d8005d00 con 0x7fc8dc071de0 2023-12-13T23:55:21.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fea3402cd50 con 0x7fea3c15f6d0 2023-12-13T23:55:21.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0538005250 con 0x7f054c049f80 2023-12-13T23:55:21.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0554072600 con 0x7f054c049f80 2023-12-13T23:55:21.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.907+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa59c076da0 con 0x7fa5a00b1880 2023-12-13T23:55:21.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7fea23fff640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 0x7fea28071a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea340aca80 con 0x7fea3c15f6d0 2023-12-13T23:55:21.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7f0542ffd640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 0x7f053406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7fea3b7fe640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 0x7fea28071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7f0551d9b640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 0x7f053406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7fea3b7fe640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 0x7fea28071a90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fea3000a810 tx=0x7fea30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.908+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa92c058010 con 0x7fa93c0b0cf0 2023-12-13T23:55:21.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.909+0000 7f0551d9b640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 0x7f053406d680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0548034000 tx=0x7f0548008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.910+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f05540d2180 con 0x7f054c049f80 2023-12-13T23:55:21.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.911+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fea34077a00 con 0x7fea3c15f6d0 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.912+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc8d800c940 con 0x7fc8dc071de0 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.913+0000 7fc8d1ffb640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 0x7fc8c006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.913+0000 7fc8e1533640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 0x7fc8c006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.914+0000 7fc8e1533640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 0x7fc8c006d6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009a70 tx=0x7fc8cc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.915+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8d8058d00 con 0x7fc8dc071de0 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.915+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f05540ce4c0 con 0x7f054c049f80 2023-12-13T23:55:21.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb6f407a530 con 0x7fb6f80067e0 2023-12-13T23:55:21.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c03e070 con 0x7f7d600392f0 2023-12-13T23:55:21.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c003610 con 0x7f7d600392f0 2023-12-13T23:55:21.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c003980 con 0x7f7d600392f0 2023-12-13T23:55:21.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7f7d6d852640 1 -- 172.21.15.124:0/2280104460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6004d3e0 con 0x7f7d600392f0 2023-12-13T23:55:21.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.917+0000 7f7d6d852640 1 -- 172.21.15.124:0/2280104460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6004d9a0 con 0x7f7d600392f0 2023-12-13T23:55:21.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.919+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d48005250 con 0x7f7d600392f0 2023-12-13T23:55:21.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.920+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc8d8054940 con 0x7fc8dc071de0 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.930+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7d5c042070 con 0x7f7d600392f0 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.931+0000 7f7d657fa640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 0x7f7d4c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.931+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d5c0ab5e0 con 0x7f7d600392f0 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.931+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7d5c076320 con 0x7f7d600392f0 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.932+0000 7f7d67fff640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 0x7f7d4c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:21.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:21.945+0000 7f7d67fff640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 0x7f7d4c06d680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7d600e0580 tx=0x7f7d54008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:22.298 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.300+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa5a00cb5d0 con 0x7fa5a00b1880 2023-12-13T23:55:22.298 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.300+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6804035fd0 con 0x7f6804185a40 2023-12-13T23:55:22.299 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.301+0000 7fa595ffb640 1 -- 172.21.15.124:0/1282468168 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa59c07a3c0 con 0x7fa5a00b1880 2023-12-13T23:55:22.299 INFO:teuthology.orchestra.run.smithi124.stdout:55834574854 2023-12-13T23:55:22.299 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.301+0000 7f68097fa640 1 -- 172.21.15.124:0/4092358519 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6804035fd0 con 0x7f6804185a40 2023-12-13T23:55:22.302 INFO:teuthology.orchestra.run.smithi124.stdout:55834574854 2023-12-13T23:55:22.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 msgr2=0x7fa58406d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 0x7fa58406d750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fa5a01e8fc0 tx=0x7fa598009040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 msgr2=0x7fa5a02024e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa59c02bf50 tx=0x7fa59c004730 comp rx=0 tx=0).stop 2023-12-13T23:55:22.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 shutdown_connections 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa58406b290 0x7fa58406d750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa5a00b47c0 0x7fa5a01e5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a0158fa0 0x7fa5a01e64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 --2- 172.21.15.124:0/1282468168 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa5a00b1880 0x7fa5a02024e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 >> 172.21.15.124:0/1282468168 conn(0x7fa5a00b0d60 msgr2=0x7fa5a00b2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 shutdown_connections 2023-12-13T23:55:22.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7fa5a76ec640 1 -- 172.21.15.124:0/1282468168 wait complete. 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.304+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 msgr2=0x7f67f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 0x7f67f006d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6804003e80 tx=0x7f6800008040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 msgr2=0x7f6804185ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f680c04f5f0 tx=0x7f680c0743c0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 shutdown_connections 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f67f006b2c0 0x7f67f006d780 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6804185a40 0x7f6804185ec0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68041888e0 0x7f68040358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 --2- 172.21.15.124:0/4092358519 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67fc005180 0x7f680404e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.305+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 >> 172.21.15.124:0/4092358519 conn(0x7f6804009fb0 msgr2=0x7f6804052dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.306+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 shutdown_connections 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.306+0000 7f6812e17640 1 -- 172.21.15.124:0/4092358519 wait complete. 2023-12-13T23:55:22.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.306+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa93c069400 con 0x7fa93c0b0cf0 2023-12-13T23:55:22.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.306+0000 7fea21ffb640 1 -- 172.21.15.124:0/1987642608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fea3c0c58f0 con 0x7fea3c15f6d0 2023-12-13T23:55:22.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.307+0000 7fa938ff9640 1 -- 172.21.15.124:0/2973236335 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa92c057ba0 con 0x7fa93c0b0cf0 2023-12-13T23:55:22.308 INFO:teuthology.orchestra.run.smithi124.stdout:55834574855 2023-12-13T23:55:22.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.307+0000 7fea23fff640 1 -- 172.21.15.124:0/1987642608 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fea3c0c58f0 con 0x7fea3c15f6d0 2023-12-13T23:55:22.309 INFO:teuthology.orchestra.run.smithi124.stdout:55834574855 2023-12-13T23:55:22.310 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 msgr2=0x7fea28071a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.310 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 0x7fea28071a90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fea3000a810 tx=0x7fea30008040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.310 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 msgr2=0x7fea3c0aec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.310 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 0x7fea3c0aec00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fea34032000 tx=0x7fea34004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 msgr2=0x7fa92806d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 0x7fa92806d6d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa930036000 tx=0x7fa930008030 comp rx=0 tx=0).stop 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 msgr2=0x7fa93c15b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa92c00c430 tx=0x7fa92c00c8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 shutdown_connections 2023-12-13T23:55:22.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 shutdown_connections 2023-12-13T23:55:22.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa92806b210 0x7fa92806d6d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea2806f5d0 0x7fea28071a90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea3c15f6d0 0x7fea3c0aec00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa93c0b2060 0x7fa93c161a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea3c0d19a0 0x7fea3c0c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa93c158690 0x7fa93c165150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 --2- 172.21.15.124:0/2973236335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa93c0b0cf0 0x7fa93c15b650 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 --2- 172.21.15.124:0/1987642608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea3c0cd6c0 0x7fea3c149210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 >> 172.21.15.124:0/2973236335 conn(0x7fa93c06a4d0 msgr2=0x7fa93c1575f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.311+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 >> 172.21.15.124:0/1987642608 conn(0x7fea3c0d0e80 msgr2=0x7fea3c15beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.312+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 shutdown_connections 2023-12-13T23:55:22.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.312+0000 7fa9428ec640 1 -- 172.21.15.124:0/2973236335 wait complete. 2023-12-13T23:55:22.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.313+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 shutdown_connections 2023-12-13T23:55:22.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.313+0000 7fea420bb640 1 -- 172.21.15.124:0/1987642608 wait complete. 2023-12-13T23:55:22.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.314+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb6f8033ab0 con 0x7fb6f80067e0 2023-12-13T23:55:22.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.315+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc8dc155970 con 0x7fc8dc071de0 2023-12-13T23:55:22.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.315+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f05380057d0 con 0x7f054c049f80 2023-12-13T23:55:22.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.315+0000 7fb6fd7fa640 1 -- 172.21.15.124:0/303151247 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb6f4035020 con 0x7fb6f80067e0 2023-12-13T23:55:22.316 INFO:teuthology.orchestra.run.smithi124.stdout:55834574855 2023-12-13T23:55:22.316 INFO:teuthology.orchestra.run.smithi124.stdout:55834574855 2023-12-13T23:55:22.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.315+0000 7fc8d1ffb640 1 -- 172.21.15.124:0/1519813092 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc8dc155970 con 0x7fc8dc071de0 2023-12-13T23:55:22.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.316+0000 7f0542ffd640 1 -- 172.21.15.124:0/3077213767 <== mon.1 v2:172.21.15.203:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0554097090 con 0x7f054c049f80 2023-12-13T23:55:22.317 INFO:teuthology.orchestra.run.smithi124.stdout:55834574854 2023-12-13T23:55:22.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.318+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 msgr2=0x7fb6e406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 0x7fb6e406d680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec000f60 tx=0x7fb6ec009040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 msgr2=0x7fb6f80e4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb6f402b910 tx=0x7fb6f40046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 msgr2=0x7fc8c006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 0x7fc8c006d6b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009a70 tx=0x7fc8cc008040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 shutdown_connections 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb6e406b1c0 0x7fb6e406d680 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb6f80eacd0 0x7fb6f80e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb6f80067e0 0x7fb6f80e4c00 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 msgr2=0x7fc8dc13b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 0x7fc8dc13b580 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800e870 tx=0x7fc8d800ed30 comp rx=0 tx=0).stop 2023-12-13T23:55:22.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 --2- 172.21.15.124:0/303151247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb6f8005e60 0x7fb6f8049940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.319+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 >> 172.21.15.124:0/303151247 conn(0x7fb6f8009f80 msgr2=0x7fb6f804db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 msgr2=0x7f053406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 shutdown_connections 2023-12-13T23:55:22.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 0x7f053406d680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0548034000 tx=0x7f0548008030 comp rx=0 tx=0).stop 2023-12-13T23:55:22.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fb706bbe640 1 -- 172.21.15.124:0/303151247 wait complete. 2023-12-13T23:55:22.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 msgr2=0x7f054c179020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0554079e00 tx=0x7f0554067f00 comp rx=0 tx=0).stop 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 shutdown_connections 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f053406b1c0 0x7f053406d680 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f054c057450 0x7f054c035c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f054c04a360 0x7f054c0356d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 --2- 172.21.15.124:0/3077213767 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f054c049f80 0x7f054c179020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 >> 172.21.15.124:0/3077213767 conn(0x7f054c020010 msgr2=0x7f054c0526b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 shutdown_connections 2023-12-13T23:55:22.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7f055359e640 1 -- 172.21.15.124:0/3077213767 wait complete. 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 shutdown_connections 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8c006b1f0 0x7fc8c006d6b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc8dc071de0 0x7fc8dc13b580 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc8dc158f90 0x7fc8dc0a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 --2- 172.21.15.124:0/1519813092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc8dc06c4c0 0x7fc8dc157830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 >> 172.21.15.124:0/1519813092 conn(0x7fc8dc0b0b10 msgr2=0x7fc8dc13e350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 shutdown_connections 2023-12-13T23:55:22.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.320+0000 7fc8e3797640 1 -- 172.21.15.124:0/1519813092 wait complete. 2023-12-13T23:55:22.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.0 2023-12-13T23:55:22.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.3 2023-12-13T23:55:22.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.330+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f7d48005500 con 0x7f7d600392f0 2023-12-13T23:55:22.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.331+0000 7f7d657fa640 1 -- 172.21.15.124:0/2280104460 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7d5c079940 con 0x7f7d600392f0 2023-12-13T23:55:22.329 INFO:teuthology.orchestra.run.smithi124.stdout:55834574854 2023-12-13T23:55:22.331 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.1 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.333+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 msgr2=0x7f7d4c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.333+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 0x7f7d4c06d680 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7d600e0580 tx=0x7f7d54008040 comp rx=0 tx=0).stop 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.333+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 msgr2=0x7f7d60033eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.333+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000c00 tx=0x7f7d5c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 shutdown_connections 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7d4c06b1c0 0x7f7d4c06d680 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7d60050400 0x7f7d600d44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7d600e0070 0x7f7d60053fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 --2- 172.21.15.124:0/2280104460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d600392f0 0x7f7d60033eb0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:22.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 >> 172.21.15.124:0/2280104460 conn(0x7f7d60055ba0 msgr2=0x7f7d60055420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:22.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.4 2023-12-13T23:55:22.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 shutdown_connections 2023-12-13T23:55:22.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:22.334+0000 7f7d52ffd640 1 -- 172.21.15.124:0/2280104460 wait complete. 2023-12-13T23:55:22.338 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.6 2023-12-13T23:55:22.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574855 for osd.2 2023-12-13T23:55:22.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.7 2023-12-13T23:55:22.344 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.5 2023-12-13T23:55:23.325 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-13T23:55:23.326 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-13T23:55:23.332 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-13T23:55:23.334 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-13T23:55:23.338 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-13T23:55:23.340 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-13T23:55:23.340 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-13T23:55:23.345 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-13T23:55:23.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.490+0000 7fb3e8911640 1 Processor -- start 2023-12-13T23:55:23.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.490+0000 7fb3e8911640 1 -- start start 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.491+0000 7fb3e8911640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb3e406d810 0x7fb3e406de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.491+0000 7fb3e8911640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e406e3c0 con 0x7fb3e4071970 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.491+0000 7fb3e8911640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e40b2240 con 0x7fb3e416b370 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.491+0000 7fb3e8911640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e416d670 con 0x7fb3e406d810 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.492+0000 7fb3e3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb3e4071970 0x7fb3e406c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43640/0 (socket says 172.21.15.124:43640) 2023-12-13T23:55:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.492+0000 7fb3e3fff640 1 -- 172.21.15.124:0/1380688857 learned_addr learned my addr 172.21.15.124:0/1380688857 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:23.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.492+0000 7fb3e37fe640 1 --2- 172.21.15.124:0/1380688857 >> v2:172.21.15.124:3300/0 conn(0x7fb3e406d810 0x7fb3e406de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.493+0000 7fb3e37fe640 1 -- 172.21.15.124:0/1380688857 >> v1:172.21.15.203:6789/0 conn(0x7fb3e416b370 legacy=0x7fb3e4071260 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-13T23:55:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.493+0000 7fb3e37fe640 1 -- 172.21.15.124:0/1380688857 >> v1:172.21.15.124:6789/0 conn(0x7fb3e4071970 legacy=0x7fb3e406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.493+0000 7fb3e37fe640 1 -- 172.21.15.124:0/1380688857 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e4170980 con 0x7fb3e406d810 2023-12-13T23:55:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7fb3e37fe640 1 --2- 172.21.15.124:0/1380688857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e406d810 0x7fb3e406de80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05b9098f5792b61 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.495+0000 7fb3e27fc640 1 -- 172.21.15.124:0/1380688857 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8030710 con 0x7fb3e406d810 2023-12-13T23:55:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.495+0000 7fb3e27fc640 1 -- 172.21.15.124:0/1380688857 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d802e050 con 0x7fb3e406d810 2023-12-13T23:55:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.495+0000 7fb3e27fc640 1 -- 172.21.15.124:0/1380688857 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8030a90 con 0x7fb3e406d810 2023-12-13T23:55:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.492+0000 7f2de3fff640 1 Processor -- start 2023-12-13T23:55:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.492+0000 7f2de3fff640 1 -- start start 2023-12-13T23:55:23.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7fb3e8911640 1 -- 172.21.15.124:0/1380688857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e406d810 msgr2=0x7fb3e406de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7fb3e8911640 1 --2- 172.21.15.124:0/1380688857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e406d810 0x7fb3e406de80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d802b790 comp rx=0 tx=0).stop 2023-12-13T23:55:23.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7fb3e8911640 1 -- 172.21.15.124:0/1380688857 shutdown_connections 2023-12-13T23:55:23.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7fb3e8911640 1 --2- 172.21.15.124:0/1380688857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e406d810 0x7fb3e406de80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7fb3e8911640 1 -- 172.21.15.124:0/1380688857 >> 172.21.15.124:0/1380688857 conn(0x7fb3e40b0c60 msgr2=0x7fb3e40b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ddc0338f0 0x7f2ddc053cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2ddc054230 0x7f2ddc04f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de3fff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ddc052cc0 con 0x7f2ddc100fe0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de3fff640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ddc034190 con 0x7f2ddc054230 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de3fff640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc038320 con 0x7f2ddc0338f0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de27fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ddc0338f0 0x7f2ddc053cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de27fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ddc0338f0 0x7f2ddc053cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60164/0 (socket says 172.21.15.124:60164) 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.494+0000 7f2de27fc640 1 -- 172.21.15.124:0/282737011 learned_addr learned my addr 172.21.15.124:0/282737011 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.495+0000 7f2de37fe640 1 --2- 172.21.15.124:0/282737011 >> v2:172.21.15.203:3300/0 conn(0x7f2ddc054230 0x7f2ddc04f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.495+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188594589 0 0) 0x7f2ddc052cc0 con 0x7f2ddc100fe0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dcc003420 con 0x7f2ddc100fe0 2023-12-13T23:55:23.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 354864339 0 0) 0x7f2dcc003420 con 0x7f2ddc100fe0 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ddc0338f0 msgr2=0x7f2ddc053cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2ddc0338f0 0x7f2ddc053cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 >> v2:172.21.15.203:3300/0 conn(0x7f2ddc054230 msgr2=0x7f2ddc04f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 --2- 172.21.15.124:0/282737011 >> v2:172.21.15.203:3300/0 conn(0x7f2ddc054230 0x7f2ddc04f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc052f30 con 0x7f2ddc100fe0 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f2dd4001670 con 0x7f2ddc100fe0 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 >> v1:172.21.15.203:6789/0 conn(0x7f2ddc100fe0 legacy=0x7f2ddc037c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc004a50 0x7f2dcc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc003660 0x7f2dcc003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2dcc01c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc034190 con 0x7f2dcc004a50 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc038320 con 0x7f2dcc003660 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dcc01ce40 con 0x7f2dcc004020 2023-12-13T23:55:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de27fc640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc003660 0x7f2dcc003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.496+0000 7f2de2ffd640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc004a50 0x7f2dcc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2dcc01c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 -- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc003660 msgr2=0x7f2dcc003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2dcc003660 0x7f2dcc003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 -- 172.21.15.124:0/282737011 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc004a50 msgr2=0x7f2dcc004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2dcc004a50 0x7f2dcc004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.497+0000 7f2de37fe640 1 -- 172.21.15.124:0/282737011 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc052f30 con 0x7f2dcc004020 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.498+0000 7fb3e8911640 1 -- 172.21.15.124:0/1380688857 shutdown_connections 2023-12-13T23:55:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.498+0000 7fb3e8911640 1 -- 172.21.15.124:0/1380688857 wait complete. 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.498+0000 7f2de37fe640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2dcc01c770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009ac0 tx=0x7f2dcc0046e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c03a2e17568e542 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.498+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd802c7a0 con 0x7f2dcc004020 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.499+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd802c900 con 0x7f2dcc004020 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.499+0000 7f2de1ffb640 1 -- 172.21.15.124:0/282737011 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd802ccd0 con 0x7f2dcc004020 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 -- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 msgr2=0x7f2dcc01c770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 --2- 172.21.15.124:0/282737011 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2dcc01c770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2dd8009ac0 tx=0x7f2dcc0046e0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de37fe640 1 -- 172.21.15.124:0/282737011 reap_dead start 2023-12-13T23:55:23.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7fb3e8911640 1 Processor -- start 2023-12-13T23:55:23.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 -- 172.21.15.124:0/282737011 shutdown_connections 2023-12-13T23:55:23.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 -- 172.21.15.124:0/282737011 >> 172.21.15.124:0/282737011 conn(0x7f2ddc009f80 msgr2=0x7f2ddc00a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7fb3e8911640 1 -- start start 2023-12-13T23:55:23.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.501+0000 7fb3e8911640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e8911640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 0x7fb3e40b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e37fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e37fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60184/0 (socket says 172.21.15.124:60184) 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e37fe640 1 -- 172.21.15.124:0/666787942 learned_addr learned my addr 172.21.15.124:0/666787942 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e2ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 0x7fb3e40b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3e416b370 0x7fb3e4159210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e415a240 con 0x7fb3e4071970 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40aa3f0 con 0x7fb3e416b370 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e41579d0 con 0x7fb3e406d810 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e37fe640 1 -- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3e416b370 msgr2=0x7fb3e4159210 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e37fe640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3e416b370 0x7fb3e4159210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.503+0000 7fb3e37fe640 1 -- 172.21.15.124:0/666787942 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 msgr2=0x7fb3e40b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 -- 172.21.15.124:0/282737011 shutdown_connections 2023-12-13T23:55:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.500+0000 7f2de3fff640 1 -- 172.21.15.124:0/282737011 wait complete. 2023-12-13T23:55:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7f2de3fff640 1 Processor -- start 2023-12-13T23:55:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.502+0000 7f2de3fff640 1 -- start start 2023-12-13T23:55:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 0x7f2ddc0f63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7fb3e37fe640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 0x7fb3e40b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de2ffd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60192/0 (socket says 172.21.15.124:60192) 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de27fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 0x7f2ddc0f63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de2ffd640 1 -- 172.21.15.124:0/2012131471 learned_addr learned my addr 172.21.15.124:0/2012131471 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ddc103690 0x7f2ddc103b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0f98f0 con 0x7f2ddc0fd050 2023-12-13T23:55:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc104050 con 0x7f2ddc103690 2023-12-13T23:55:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ddc0f6930 con 0x7f2dcc004020 2023-12-13T23:55:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7f2de37fe640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ddc103690 0x7f2ddc103b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.504+0000 7fb3e37fe640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8009400 con 0x7fb3e406d810 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 -- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ddc103690 msgr2=0x7f2ddc103b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ddc103690 0x7f2ddc103b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 -- 172.21.15.124:0/2012131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 msgr2=0x7f2ddc0f63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 0x7f2ddc0f63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ddc0f8ce0 con 0x7f2dcc004020 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de27fc640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 0x7f2ddc0f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de2ffd640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400fb60 tx=0x7f2dd400d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7fb3e2ffd640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 0x7fb3e40b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd400fe60 con 0x7f2dcc004020 2023-12-13T23:55:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7fb3e37fe640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802bda0 tx=0x7fb3d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ddc0f8fc0 con 0x7f2dcc004020 2023-12-13T23:55:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.505+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ddc0f9580 con 0x7f2dcc004020 2023-12-13T23:55:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd400dc30 con 0x7f2dcc004020 2023-12-13T23:55:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd40050d0 con 0x7f2dcc004020 2023-12-13T23:55:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80329d0 con 0x7fb3e406d810 2023-12-13T23:55:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d8032b30 con 0x7fb3e406d810 2023-12-13T23:55:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e4159f60 con 0x7fb3e406d810 2023-12-13T23:55:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.507+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8032ea0 con 0x7fb3e406d810 2023-12-13T23:55:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.508+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e4157660 con 0x7fb3e406d810 2023-12-13T23:55:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.508+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ddc0089a0 con 0x7f2dcc004020 2023-12-13T23:55:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.509+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2dd4005230 con 0x7f2dcc004020 2023-12-13T23:55:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.510+0000 7f2dcb7fe640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 0x7f2dd006d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.510+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2dd40899d0 con 0x7f2dcc004020 2023-12-13T23:55:23.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.511+0000 7f2de27fc640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 0x7f2dd006d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.511+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb3d8039040 con 0x7fb3e406d810 2023-12-13T23:55:23.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.511+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e41571e0 con 0x7fb3e406d810 2023-12-13T23:55:23.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.511+0000 7fb3cbfff640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 0x7fb3d006d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.512+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb3d80ab970 con 0x7fb3e406d810 2023-12-13T23:55:23.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.512+0000 7fb3e2ffd640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 0x7fb3d006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.512+0000 7f2de27fc640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 0x7f2dd006d6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2de40620a0 tx=0x7f2de404f730 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.512+0000 7fb3e2ffd640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 0x7fb3d006d750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40cb0a0 tx=0x7fb3d4009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.515+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2dd4054920 con 0x7f2dcc004020 2023-12-13T23:55:23.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.519+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb3e41571e0 con 0x7fb3e406d810 2023-12-13T23:55:23.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.541+0000 7f652e0bb640 1 Processor -- start 2023-12-13T23:55:23.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.541+0000 7f652e0bb640 1 -- start start 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.542+0000 7f652e0bb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f65280b1780 0x7f65280b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.542+0000 7f652e0bb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f65281597f0 0x7f6528071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.542+0000 7f652e0bb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65280b2170 con 0x7f6528156d40 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.542+0000 7f652e0bb640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f652815a7f0 con 0x7f65281597f0 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.542+0000 7f652e0bb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65281571a0 con 0x7f65280b1780 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f65281597f0 0x7f6528071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f65281597f0 0x7f6528071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57356/0 (socket says 172.21.15.124:57356) 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 -- 172.21.15.124:0/1192643209 learned_addr learned my addr 172.21.15.124:0/1192643209 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d0b9640 1 --2- 172.21.15.124:0/1192643209 >> v2:172.21.15.203:3300/0 conn(0x7f65280b1780 0x7f65280b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 -- 172.21.15.124:0/1192643209 >> v2:172.21.15.203:3300/0 conn(0x7f65280b1780 msgr2=0x7f65280b3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 --2- 172.21.15.124:0/1192643209 >> v2:172.21.15.203:3300/0 conn(0x7f65280b1780 0x7f65280b3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 -- 172.21.15.124:0/1192643209 >> v1:172.21.15.124:6789/0 conn(0x7f6528156d40 legacy=0x7f65280cc950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.543+0000 7f652d8ba640 1 -- 172.21.15.124:0/1192643209 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6528159f50 con 0x7f65281597f0 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.544+0000 7f652d8ba640 1 --2- 172.21.15.124:0/1192643209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65281597f0 0x7f6528071260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f652806ff20 tx=0x7f651802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f3cf204f7784f34 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.544+0000 7f651ffff640 1 -- 172.21.15.124:0/1192643209 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518030860 con 0x7f65281597f0 2023-12-13T23:55:23.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.544+0000 7f651ffff640 1 -- 172.21.15.124:0/1192643209 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65180309c0 con 0x7f65281597f0 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.544+0000 7f651ffff640 1 -- 172.21.15.124:0/1192643209 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518030d50 con 0x7f65281597f0 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.545+0000 7f652e0bb640 1 -- 172.21.15.124:0/1192643209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65281597f0 msgr2=0x7f6528071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.545+0000 7f652e0bb640 1 --2- 172.21.15.124:0/1192643209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65281597f0 0x7f6528071260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f652806ff20 tx=0x7f651802b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f652e0bb640 1 -- 172.21.15.124:0/1192643209 shutdown_connections 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f652e0bb640 1 --2- 172.21.15.124:0/1192643209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f65281597f0 0x7f6528071260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f652e0bb640 1 --2- 172.21.15.124:0/1192643209 >> v2:172.21.15.203:3300/0 conn(0x7f65280b1780 0x7f65280b3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f652e0bb640 1 -- 172.21.15.124:0/1192643209 >> 172.21.15.124:0/1192643209 conn(0x7f65280b0c60 msgr2=0x7f65280b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7fc03b2bb640 1 Processor -- start 2023-12-13T23:55:23.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f92f9c59640 1 Processor -- start 2023-12-13T23:55:23.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f92f9c59640 1 -- start start 2023-12-13T23:55:23.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7fc03b2bb640 1 -- start start 2023-12-13T23:55:23.545 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.546+0000 7f652e0bb640 1 -- 172.21.15.124:0/1192643209 shutdown_connections 2023-12-13T23:55:23.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.547+0000 7fc03b2bb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc034158690 0x7fc0340b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.547+0000 7f92f9c59640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 0x7f92ec053bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f9c59640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f92ec054100 0x7f92ec0068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc039057640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc034158690 0x7fc0340b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.547+0000 7f652e0bb640 1 -- 172.21.15.124:0/1192643209 wait complete. 2023-12-13T23:55:23.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc039057640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc034158690 0x7fc0340b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57364/0 (socket says 172.21.15.124:57364) 2023-12-13T23:55:23.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc039057640 1 -- 172.21.15.124:0/3101158190 learned_addr learned my addr 172.21.15.124:0/3101158190 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0340b0cf0 0x7fc0340b1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 0x7f92ec053bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 0x7f92ec053bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51696/0 (socket says 172.21.15.124:51696) 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f3fff640 1 -- 172.21.15.124:0/918786562 learned_addr learned my addr 172.21.15.124:0/918786562 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f9458640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f92ec054100 0x7f92ec0068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ec058540 con 0x7f92ec0e3090 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ec006170 con 0x7f92ec054100 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ec054570 con 0x7f92ec03e810 2023-12-13T23:55:23.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f92f8c57640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f92ec0e3090 0x7f92ec057d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43656/0 (socket says 172.21.15.124:43656) 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> v2:172.21.15.203:3300/0 conn(0x7fc0341651b0 0x7fc034165640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc034070390 con 0x7fc034158690 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc034162d80 con 0x7fc0341651b0 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0340b1690 con 0x7fc0340b0cf0 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f5e7ac7c640 1 Processor -- start 2023-12-13T23:55:23.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7f5e7ac7c640 1 -- start start 2023-12-13T23:55:23.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.548+0000 7fc039858640 1 --2- 172.21.15.124:0/3101158190 >> v2:172.21.15.203:3300/0 conn(0x7fc0341651b0 0x7fc034165640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc038856640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0340b0cf0 0x7fc0340b1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f652e0bb640 1 Processor -- start 2023-12-13T23:55:23.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f652e0bb640 1 -- start start 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc039057640 1 -- 172.21.15.124:0/3101158190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0340b0cf0 msgr2=0x7fc0340b1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc039057640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0340b0cf0 0x7fc0340b1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc039057640 1 -- 172.21.15.124:0/3101158190 >> v2:172.21.15.203:3300/0 conn(0x7fc0341651b0 msgr2=0x7fc034165640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f92f9458640 1 -- 172.21.15.124:0/918786562 >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 msgr2=0x7f92ec053bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f92f9458640 1 --2- 172.21.15.124:0/918786562 >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 0x7f92ec053bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f92f9458640 1 -- 172.21.15.124:0/918786562 >> v1:172.21.15.124:6789/0 conn(0x7f92ec0e3090 legacy=0x7f92ec057d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc039057640 1 --2- 172.21.15.124:0/3101158190 >> v2:172.21.15.203:3300/0 conn(0x7fc0341651b0 0x7fc034165640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f92f9458640 1 -- 172.21.15.124:0/918786562 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ec058820 con 0x7f92ec054100 2023-12-13T23:55:23.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7fc039057640 1 -- 172.21.15.124:0/3101158190 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0340705a0 con 0x7fc034158690 2023-12-13T23:55:23.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652e0bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 0x7f6528142c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c04ffa0 0x7f5e6c037ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5e6c0380e0 0x7f5e6c038570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e78a18640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e78a18640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57382/0 (socket says 172.21.15.124:57382) 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e78a18640 1 -- 172.21.15.124:0/2923170986 learned_addr learned my addr 172.21.15.124:0/2923170986 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c054300 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c0536f0 con 0x7f5e6c0380e0 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0539d0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.549+0000 7f5e79219640 1 --2- 172.21.15.124:0/2923170986 >> v2:172.21.15.203:3300/0 conn(0x7f5e6c0380e0 0x7f5e6c038570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f5e78a18640 1 -- 172.21.15.124:0/2923170986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c04ffa0 msgr2=0x7f5e6c037ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f5e78a18640 1 --2- 172.21.15.124:0/2923170986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c04ffa0 0x7f5e6c037ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f5e78a18640 1 -- 172.21.15.124:0/2923170986 >> v2:172.21.15.203:3300/0 conn(0x7f5e6c0380e0 msgr2=0x7f5e6c038570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f5e78a18640 1 --2- 172.21.15.124:0/2923170986 >> v2:172.21.15.203:3300/0 conn(0x7f5e6c0380e0 0x7f5e6c038570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f5e78a18640 1 -- 172.21.15.124:0/2923170986 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c053cb0 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652e0bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 0x7f652813f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652d0b9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 0x7f6528142c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f92f9458640 1 --2- 172.21.15.124:0/918786562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec0068a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f92ec008650 tx=0x7f92e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d696c36b2db35c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652d0b9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 0x7f6528142c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51730/0 (socket says 172.21.15.124:51730) 2023-12-13T23:55:23.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652d0b9640 1 -- 172.21.15.124:0/3102164762 learned_addr learned my addr 172.21.15.124:0/3102164762 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f652e0bb640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 0x7f652813f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f92f37fe640 1 -- 172.21.15.124:0/918786562 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8030860 con 0x7f92ec054100 2023-12-13T23:55:23.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f92f37fe640 1 -- 172.21.15.124:0/918786562 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e80309c0 con 0x7f92ec054100 2023-12-13T23:55:23.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f92f37fe640 1 -- 172.21.15.124:0/918786562 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e8030d50 con 0x7f92ec054100 2023-12-13T23:55:23.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7fc039057640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc034158690 0x7fc0340b44f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc03415c200 tx=0x7fc03002bb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=661e242af13a266c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f5e78a18640 1 --2- 172.21.15.124:0/2923170986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00bcd0 tx=0x7f5e6803c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6447aeefabf772 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7fc02bfff640 1 -- 172.21.15.124:0/3101158190 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03002c730 con 0x7fc034158690 2023-12-13T23:55:23.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7fc02bfff640 1 -- 172.21.15.124:0/3101158190 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03002c890 con 0x7fc034158690 2023-12-13T23:55:23.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7fc02bfff640 1 -- 172.21.15.124:0/3101158190 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03002cc00 con 0x7fc034158690 2023-12-13T23:55:23.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f8da58ea640 1 Processor -- start 2023-12-13T23:55:23.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f652c8b8640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 0x7f652813f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652e0bb640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6528158df0 con 0x7f6528156d40 2023-12-13T23:55:23.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652e0bb640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f652813ffd0 con 0x7f65280b1780 2023-12-13T23:55:23.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f5e737fe640 1 -- 172.21.15.124:0/2923170986 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68041890 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f5e737fe640 1 -- 172.21.15.124:0/2923170986 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e680419f0 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652e0bb640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6528143150 con 0x7f65281597f0 2023-12-13T23:55:23.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f5e737fe640 1 -- 172.21.15.124:0/2923170986 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e68041dc0 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 0x7f652813f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 msgr2=0x7f92ec0068a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 --2- 172.21.15.124:0/918786562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec0068a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f92ec008650 tx=0x7f92e802b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fb001575640 1 Processor -- start 2023-12-13T23:55:23.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.550+0000 7f8da58ea640 1 -- start start 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da58ea640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da58ea640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8da006c560 0x7f8da006c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da58ea640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da016e780 con 0x7f8da0071260 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da58ea640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da00b2aa0 con 0x7f8da006c560 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da58ea640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00b2ec0 con 0x7f8da016d9e0 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8d9ffff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8d9ffff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60212/0 (socket says 172.21.15.124:60212) 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8d9ffff640 1 -- 172.21.15.124:0/234857759 learned_addr learned my addr 172.21.15.124:0/234857759 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.551+0000 7f8da50e9640 1 --2- 172.21.15.124:0/234857759 >> v2:172.21.15.203:3300/0 conn(0x7f8da006c560 0x7f8da006c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f8d9f7fe640 1 -- 172.21.15.124:0/234857759 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 384319463 0 0) 0x7f8da016e780 con 0x7f8da0071260 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f8d9ffff640 1 -- 172.21.15.124:0/234857759 >> v2:172.21.15.203:3300/0 conn(0x7f8da006c560 msgr2=0x7f8da006c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f8d9ffff640 1 --2- 172.21.15.124:0/234857759 >> v2:172.21.15.203:3300/0 conn(0x7f8da006c560 0x7f8da006c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f8d9ffff640 1 -- 172.21.15.124:0/234857759 >> v1:172.21.15.203:6789/0 conn(0x7f8da0071260 legacy=0x7f8da0071640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f8d9ffff640 1 -- 172.21.15.124:0/234857759 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da016b0f0 con 0x7f8da016d9e0 2023-12-13T23:55:23.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc034158690 msgr2=0x7fc0340b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc034158690 0x7fc0340b44f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc03415c200 tx=0x7fc03002bb90 comp rx=0 tx=0).stop 2023-12-13T23:55:23.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 shutdown_connections 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 shutdown_connections 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 --2- 172.21.15.124:0/918786562 >> v2:172.21.15.203:3300/0 conn(0x7f92ec03e810 0x7f92ec053bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 --2- 172.21.15.124:0/918786562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec0068a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> v2:172.21.15.203:3300/0 conn(0x7fc0341651b0 0x7fc034165640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0340b0cf0 0x7fc0340b1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/3101158190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc034158690 0x7fc0340b44f0 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fc03415c200 tx=0x7fc03002bb90 comp rx=0 tx=0).stop 2023-12-13T23:55:23.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 >> 172.21.15.124:0/3101158190 conn(0x7fc03406a4d0 msgr2=0x7fc03406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 >> 172.21.15.124:0/918786562 conn(0x7f92ec0559b0 msgr2=0x7f92ec055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 -- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 msgr2=0x7f6528142c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 0x7f6528142c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 -- 172.21.15.124:0/3102164762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 msgr2=0x7f652813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 0x7f652813f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65180094d0 con 0x7f65281597f0 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652d8ba640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 0x7f652813f630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6518033000 tx=0x7f6518004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7f652c8b8640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 0x7f652813f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518039a90 con 0x7f65281597f0 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6518039bf0 con 0x7f65281597f0 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65180433f0 con 0x7f65281597f0 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c0eacd0 msgr2=0x7f5e6c04fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/2923170986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c00bcd0 tx=0x7f5e6803c9d0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.552+0000 7fb001575640 1 -- start start 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 shutdown_connections 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/2923170986 >> v2:172.21.15.203:3300/0 conn(0x7f5e6c0380e0 0x7f5e6c038570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/2923170986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c04ffa0 0x7f5e6c037ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/2923170986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c0eacd0 0x7f5e6c04fa60 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 >> 172.21.15.124:0/2923170986 conn(0x7f5e6c009f80 msgr2=0x7f5e6c00a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f8d9ffff640 1 --2- 172.21.15.124:0/234857759 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8d98008e10 tx=0x7f8d98034830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df672ab95eb2bbc4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 shutdown_connections 2023-12-13T23:55:23.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f652e0bb640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6528158270 con 0x7f65281597f0 2023-12-13T23:55:23.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f92f9c59640 1 -- 172.21.15.124:0/918786562 wait complete. 2023-12-13T23:55:23.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f652e0bb640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6528158830 con 0x7f65281597f0 2023-12-13T23:55:23.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7fb001575640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7f8d9f7fe640 1 -- 172.21.15.124:0/234857759 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9803a790 con 0x7f8da016d9e0 2023-12-13T23:55:23.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7f8d9f7fe640 1 -- 172.21.15.124:0/234857759 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9803a8f0 con 0x7f8da016d9e0 2023-12-13T23:55:23.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7f8d9f7fe640 1 -- 172.21.15.124:0/234857759 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9803ac80 con 0x7f8da016d9e0 2023-12-13T23:55:23.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.553+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 shutdown_connections 2023-12-13T23:55:23.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 shutdown_connections 2023-12-13T23:55:23.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7fc03b2bb640 1 -- 172.21.15.124:0/3101158190 wait complete. 2023-12-13T23:55:23.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7fb001575640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc0b2170 con 0x7faffc071260 2023-12-13T23:55:23.569 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 -- 172.21.15.124:0/234857759 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 msgr2=0x7f8da016de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 --2- 172.21.15.124:0/234857759 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f8d98008e10 tx=0x7f8d98034830 comp rx=0 tx=0).stop 2023-12-13T23:55:23.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7fb001575640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faffc1571b0 con 0x7faffc1597f0 2023-12-13T23:55:23.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7fb001575640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc15a7f0 con 0x7faffc156dd0 2023-12-13T23:55:23.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.554+0000 7faffbfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7faffbfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60242/0 (socket says 172.21.15.124:60242) 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7faffbfff640 1 -- 172.21.15.124:0/1329946968 learned_addr learned my addr 172.21.15.124:0/1329946968 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f92f9c59640 1 Processor -- start 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f92f9c59640 1 -- start start 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7faffbfff640 1 -- 172.21.15.124:0/1329946968 >> v1:172.21.15.203:6789/0 conn(0x7faffc1597f0 legacy=0x7faffc0cc950 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-12-13T23:55:23.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7faffbfff640 1 -- 172.21.15.124:0/1329946968 >> v1:172.21.15.124:6789/0 conn(0x7faffc071260 legacy=0x7faffc0b2600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/2923170986 wait complete. 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 -- 172.21.15.124:0/234857759 shutdown_connections 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 --2- 172.21.15.124:0/234857759 >> v2:172.21.15.203:3300/0 conn(0x7f8da006c560 0x7f8da006c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 --2- 172.21.15.124:0/234857759 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da016d9e0 0x7f8da016de70 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 -- 172.21.15.124:0/234857759 >> 172.21.15.124:0/234857759 conn(0x7f8da00b0c60 msgr2=0x7f8da00b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 -- 172.21.15.124:0/234857759 shutdown_connections 2023-12-13T23:55:23.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f8da58ea640 1 -- 172.21.15.124:0/234857759 wait complete. 2023-12-13T23:55:23.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6504005250 con 0x7f65281597f0 2023-12-13T23:55:23.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.555+0000 7faffbfff640 1 -- 172.21.15.124:0/1329946968 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc159f50 con 0x7faffc156dd0 2023-12-13T23:55:23.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.556+0000 7fc03b2bb640 1 Processor -- start 2023-12-13T23:55:23.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7fc03b2bb640 1 -- start start 2023-12-13T23:55:23.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f5e7ac7c640 1 Processor -- start 2023-12-13T23:55:23.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.556+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6518043550 con 0x7f65281597f0 2023-12-13T23:55:23.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.556+0000 7f651dffb640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 0x7f650c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65180abb90 con 0x7f65281597f0 2023-12-13T23:55:23.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7faffbfff640 1 --2- 172.21.15.124:0/1329946968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faff00098a0 tx=0x7faff002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68550085a80f8e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f8da58ea640 1 Processor -- start 2023-12-13T23:55:23.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f8da58ea640 1 -- start start 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f9c59640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f9c59640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f8c57640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57412/0 (socket says 172.21.15.124:57412) 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f8c57640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60256/0 (socket says 172.21.15.124:60256) 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f3fff640 1 -- 172.21.15.124:0/1974052054 learned_addr learned my addr 172.21.15.124:0/1974052054 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 0x7f92ec0d5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec0d4430 con 0x7f92ec054100 2023-12-13T23:55:23.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec0da630 con 0x7f92ec0e3090 2023-12-13T23:55:23.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.557+0000 7f5e7ac7c640 1 -- start start 2023-12-13T23:55:23.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ec0da8d0 con 0x7f92ec03e810 2023-12-13T23:55:23.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f9458640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 0x7f92ec0d5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 0x7fc034156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 0x7fc03415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc034159bf0 con 0x7fc0341502c0 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc03415b540 con 0x7fc0340b0cf0 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc03b2bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03415b820 con 0x7fc0341651b0 2023-12-13T23:55:23.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 0x7f8da00aa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8da016d9e0 0x7f8da00c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00aa060 con 0x7f8da016d9e0 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00c4a10 con 0x7f8da0071260 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da58ea640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00c4cf0 con 0x7f8da006c560 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f652d0b9640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 0x7f650c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc039858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc039858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57422/0 (socket says 172.21.15.124:57422) 2023-12-13T23:55:23.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc039858640 1 -- 172.21.15.124:0/2813836626 learned_addr learned my addr 172.21.15.124:0/2813836626 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc039057640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 0x7fc034156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 0x7f5e6c1742f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f3fff640 1 -- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 msgr2=0x7f92ec0d3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7fc038856640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 0x7fc03415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da48e8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8d9ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 0x7f8da00aa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8d9ffff640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 0x7f8da00aa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:51754/0 (socket says 172.21.15.124:51754) 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8da48e8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60272/0 (socket says 172.21.15.124:60272) 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f8d9ffff640 1 -- 172.21.15.124:0/2990118647 learned_addr learned my addr 172.21.15.124:0/2990118647 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c0eacd0 0x7f5e6c0e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f3fff640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f3fff640 1 -- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 msgr2=0x7f92ec0d5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f3fff640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 0x7f92ec0d5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f3fff640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92e8009400 con 0x7f92ec054100 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0e4620 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c174830 con 0x7f5e6c0380e0 2023-12-13T23:55:23.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e7ac7c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c049630 con 0x7f5e6c0eacd0 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e73fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e73fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57440/0 (socket says 172.21.15.124:57440) 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 -- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 msgr2=0x7fc03415acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 0x7fc03415acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 -- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 msgr2=0x7fc034156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 0x7fc034156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 -- 172.21.15.124:0/2813836626 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300094d0 con 0x7fc0341502c0 2023-12-13T23:55:23.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e73fff640 1 -- 172.21.15.124:0/1796109554 learned_addr learned my addr 172.21.15.124:0/1796109554 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f9458640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 0x7f92ec0d5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:23.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039057640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 0x7fc034156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:23.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc038856640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 0x7fc03415acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:23.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f92f8c57640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:23.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.558+0000 7f5e79219640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c0eacd0 0x7f5e6c0e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e78a18640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 0x7f5e6c1742f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc039858640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007e60 tx=0x7fc02c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 -- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c0eacd0 msgr2=0x7f5e6c0e85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c0eacd0 0x7f5e6c0e85c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da50e9640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8da016d9e0 0x7f8da00c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 -- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 msgr2=0x7f8da00aa820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 0x7f8da00aa820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 -- 172.21.15.124:0/2990118647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8da016d9e0 msgr2=0x7f8da00c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8da016d9e0 0x7f8da00c3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 -- 172.21.15.124:0/2990118647 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98008a70 con 0x7f8da006c560 2023-12-13T23:55:23.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 -- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 msgr2=0x7f5e6c1742f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 0x7f5e6c1742f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f8da48e8640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c450 tx=0x7f8d9000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 -- 172.21.15.124:0/1796109554 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e680094d0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014420 con 0x7fc0341502c0 2023-12-13T23:55:23.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e78a18640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 0x7f5e6c1742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:23.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e73fff640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000b450 tx=0x7f5e6000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c0145a0 con 0x7fc0341502c0 2023-12-13T23:55:23.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f92f3fff640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d7d0 tx=0x7f92e400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc034159e90 con 0x7fc0341502c0 2023-12-13T23:55:23.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60011890 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c014950 con 0x7fc0341502c0 2023-12-13T23:55:23.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e600119f0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f652d0b9640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 0x7f650c06d680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6524006fd0 tx=0x7f6524008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c048d90 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03415b230 con 0x7fc0341502c0 2023-12-13T23:55:23.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.559+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60011da0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c049350 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e40096a0 con 0x7f92ec054100 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92e4009800 con 0x7f92ec054100 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92e4009bb0 con 0x7f92ec054100 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.560+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ec0dabb0 con 0x7f92ec054100 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7faffaffd640 1 -- 172.21.15.124:0/1329946968 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0031640 con 0x7faffc156dd0 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7faffaffd640 1 -- 172.21.15.124:0/1329946968 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff00317a0 con 0x7faffc156dd0 2023-12-13T23:55:23.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7faffaffd640 1 -- 172.21.15.124:0/1329946968 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0031b30 con 0x7faffc156dd0 2023-12-13T23:55:23.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900035f0 con 0x7f8da006c560 2023-12-13T23:55:23.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d90003750 con 0x7f8da006c560 2023-12-13T23:55:23.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.561+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ec0db170 con 0x7f92ec054100 2023-12-13T23:55:23.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90003b00 con 0x7f8da006c560 2023-12-13T23:55:23.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7fb001575640 1 -- 172.21.15.124:0/1329946968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 msgr2=0x7faffc0b3f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da00aa300 con 0x7f8da006c560 2023-12-13T23:55:23.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7fb001575640 1 --2- 172.21.15.124:0/1329946968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faff00098a0 tx=0x7faff002b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6518076960 con 0x7f65281597f0 2023-12-13T23:55:23.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7fb001575640 1 -- 172.21.15.124:0/1329946968 shutdown_connections 2023-12-13T23:55:23.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da0214840 con 0x7f8da006c560 2023-12-13T23:55:23.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7fb001575640 1 --2- 172.21.15.124:0/1329946968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc0b3f80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7fb001575640 1 -- 172.21.15.124:0/1329946968 >> 172.21.15.124:0/1329946968 conn(0x7faffc0b0c60 msgr2=0x7faffc0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.562+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c005360 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc02c014ab0 con 0x7fc0341502c0 2023-12-13T23:55:23.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5e6001a400 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f5e717fa640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 0x7f5e5806d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f92e4009d10 con 0x7f92ec054100 2023-12-13T23:55:23.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f92f17fa640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 0x7f92e006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e60089cf0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f92e408a280 con 0x7f92ec054100 2023-12-13T23:55:23.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.564+0000 7f92f8c57640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 0x7f92e006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7fc029ffb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 0x7fc01806d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.564+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da0069320 con 0x7f8da006c560 2023-12-13T23:55:23.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.563+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc02c08a280 con 0x7fc0341502c0 2023-12-13T23:55:23.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.564+0000 7fb001575640 1 -- 172.21.15.124:0/1329946968 shutdown_connections 2023-12-13T23:55:23.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.564+0000 7fb001575640 1 -- 172.21.15.124:0/1329946968 wait complete. 2023-12-13T23:55:23.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.564+0000 7f92f8c57640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 0x7f92e006d680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f92f404f5f0 tx=0x7f92f406d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7fc039057640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 0x7fc01806d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8d90003c60 con 0x7f8da006c560 2023-12-13T23:55:23.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f5e78a18640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 0x7f5e5806d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f8d9d7fa640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 0x7f8d8c06d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ec0057b0 con 0x7f92ec054100 2023-12-13T23:55:23.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f8d9ffff640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 0x7f8d8c06d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.565+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d90088f30 con 0x7f8da006c560 2023-12-13T23:55:23.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7f5e78a18640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 0x7f5e5806d6e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e68047000 tx=0x7f5e6804c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7fc039057640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 0x7fc01806d6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc030036000 tx=0x7fc030030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7fb001575640 1 Processor -- start 2023-12-13T23:55:23.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03415a0d0 con 0x7fc0341502c0 2023-12-13T23:55:23.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7fb001575640 1 -- start start 2023-12-13T23:55:23.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.566+0000 7f8d9ffff640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 0x7f8d8c06d750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8da0213a40 tx=0x7f8d9803a000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc1597f0 0x7faffc14eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7faffbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc159240 con 0x7faffc071260 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1586c0 con 0x7faffc1597f0 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb001575640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faffc1589a0 con 0x7faffc156dd0 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7faffbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57446/0 (socket says 172.21.15.124:57446) 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7faffbfff640 1 -- 172.21.15.124:0/3438315089 learned_addr learned my addr 172.21.15.124:0/3438315089 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7faffb7fe640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.567+0000 7fb000d74640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc1597f0 0x7faffc14eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 -- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 msgr2=0x7faffc157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 -- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc1597f0 msgr2=0x7faffc14eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc1597f0 0x7faffc14eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 -- 172.21.15.124:0/3438315089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff00094d0 con 0x7faffc071260 2023-12-13T23:55:23.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.568+0000 7faffbfff640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faff00097f0 tx=0x7faff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.569+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8d90053d80 con 0x7f8da006c560 2023-12-13T23:55:23.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.569+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5e60054990 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.605 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.576+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f92e4054fa0 con 0x7f92ec054100 2023-12-13T23:55:23.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.585+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff003ad30 con 0x7faffc071260 2023-12-13T23:55:23.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.585+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff00443f0 con 0x7faffc071260 2023-12-13T23:55:23.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.585+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff00447a0 con 0x7faffc071260 2023-12-13T23:55:23.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.585+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc03415a0d0 con 0x7fc0341502c0 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.586+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faffc158c80 con 0x7faffc071260 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.586+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faffc158380 con 0x7faffc071260 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.589+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7faff002f050 con 0x7faffc071260 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.589+0000 7faff8ff9640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 0x7fafe006d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.589+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faff00aba00 con 0x7faffc071260 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.590+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faffc069320 con 0x7faffc071260 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.591+0000 7faffb7fe640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 0x7fafe006d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:23.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.600+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7faff0076850 con 0x7faffc071260 2023-12-13T23:55:23.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.614+0000 7faffb7fe640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 0x7fafe006d750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faffc1573f0 tx=0x7fafe8009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:23.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.909+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2ddc0f9a60 con 0x7f2dcc004020 2023-12-13T23:55:23.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.910+0000 7f2dcb7fe640 1 -- 172.21.15.124:0/2012131471 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2dd4057f40 con 0x7f2dcc004020 2023-12-13T23:55:23.909 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:23.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb3e41571e0 con 0x7fb3e406d810 2023-12-13T23:55:23.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7fb3cbfff640 1 -- 172.21.15.124:0/666787942 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3e41571e0 con 0x7fb3e406d810 2023-12-13T23:55:23.911 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:23.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 msgr2=0x7f2dd006d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 0x7f2dd006d6d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f2de40620a0 tx=0x7f2de404f730 comp rx=0 tx=0).stop 2023-12-13T23:55:23.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 msgr2=0x7f2ddc0f8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.913+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2dd400fb60 tx=0x7f2dd400d520 comp rx=0 tx=0).stop 2023-12-13T23:55:23.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 shutdown_connections 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2dd006b210 0x7f2dd006d6d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2ddc103690 0x7f2ddc103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2ddc0fd050 0x7f2ddc0f63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 --2- 172.21.15.124:0/2012131471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2dcc004020 0x7f2ddc0f8620 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 >> 172.21.15.124:0/2012131471 conn(0x7f2ddc009f80 msgr2=0x7f2ddc036a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 shutdown_connections 2023-12-13T23:55:23.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.914+0000 7f2de3fff640 1 -- 172.21.15.124:0/2012131471 wait complete. 2023-12-13T23:55:23.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 msgr2=0x7fb3d006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 0x7fb3d006d750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40cb0a0 tx=0x7fb3d4009060 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 msgr2=0x7fb3e40afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3d802bda0 tx=0x7fb3d80046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 shutdown_connections 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.916+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb3d006b290 0x7fb3d006d750 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40cb0a0 tx=0x7fb3d4009060 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3e416b370 0x7fb3e4159210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb3e4071970 0x7fb3e40b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 --2- 172.21.15.124:0/666787942 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3e406d810 0x7fb3e40afc90 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 >> 172.21.15.124:0/666787942 conn(0x7fb3e40b0c60 msgr2=0x7fb3e416ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 shutdown_connections 2023-12-13T23:55:23.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.917+0000 7fb3e8911640 1 -- 172.21.15.124:0/666787942 wait complete. 2023-12-13T23:55:23.930 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.3 2023-12-13T23:55:23.930 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.0 2023-12-13T23:55:23.931 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.949+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8da016cfb0 con 0x7f8da006c560 2023-12-13T23:55:23.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.950+0000 7f8d9d7fa640 1 -- 172.21.15.124:0/2990118647 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8d90017090 con 0x7f8da006c560 2023-12-13T23:55:23.948 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 msgr2=0x7f8d8c06d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 0x7f8d8c06d750 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8da0213a40 tx=0x7f8d9803a000 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 msgr2=0x7f8da020db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c450 tx=0x7f8d9000c910 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 shutdown_connections 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8c06b290 0x7f8d8c06d750 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8da0071260 0x7f8da00aa820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8da016d9e0 0x7f8da00c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 --2- 172.21.15.124:0/2990118647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8da006c560 0x7f8da020db00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 >> 172.21.15.124:0/2990118647 conn(0x7f8da00b0c60 msgr2=0x7f8da00af910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 shutdown_connections 2023-12-13T23:55:23.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.953+0000 7f8da58ea640 1 -- 172.21.15.124:0/2990118647 wait complete. 2023-12-13T23:55:23.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.964+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6504005500 con 0x7f65281597f0 2023-12-13T23:55:23.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.965+0000 7f651dffb640 1 -- 172.21.15.124:0/3102164762 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6518079f80 con 0x7f65281597f0 2023-12-13T23:55:23.963 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:23.966 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.7 2023-12-13T23:55:23.967 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.967+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 msgr2=0x7f650c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 0x7f650c06d680 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6524006fd0 tx=0x7f6524008040 comp rx=0 tx=0).stop 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 msgr2=0x7f652813f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 0x7f652813f630 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6518033000 tx=0x7f6518004730 comp rx=0 tx=0).stop 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 shutdown_connections 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f650c06b1c0 0x7f650c06d680 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f65281597f0 0x7f652813f630 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6528156d40 0x7f652813f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 --2- 172.21.15.124:0/3102164762 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f65280b1780 0x7f6528142c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 >> 172.21.15.124:0/3102164762 conn(0x7f65280b0c60 msgr2=0x7f65280af110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 shutdown_connections 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.968+0000 7f650b7fe640 1 -- 172.21.15.124:0/3102164762 wait complete. 2023-12-13T23:55:23.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.971+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f5e6c0e4760 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.971+0000 7f5e717fa640 1 -- 172.21.15.124:0/1796109554 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5e60057fb0 con 0x7f5e6c04ffa0 2023-12-13T23:55:23.969 INFO:teuthology.orchestra.run.smithi124.stdout:55834574856 2023-12-13T23:55:23.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 msgr2=0x7f5e5806d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 0x7f5e5806d6e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f5e68047000 tx=0x7f5e6804c040 comp rx=0 tx=0).stop 2023-12-13T23:55:23.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 msgr2=0x7f5e6c0e3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000b450 tx=0x7f5e6000b910 comp rx=0 tx=0).stop 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 shutdown_connections 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e5806b220 0x7f5e5806d6e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e6c0eacd0 0x7f5e6c0e85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e6c04ffa0 0x7f5e6c0e3cf0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 --2- 172.21.15.124:0/1796109554 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e6c0380e0 0x7f5e6c1742f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.974+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 >> 172.21.15.124:0/1796109554 conn(0x7f5e6c009f80 msgr2=0x7f5e6c049e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.975+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 shutdown_connections 2023-12-13T23:55:23.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.975+0000 7f5e7ac7c640 1 -- 172.21.15.124:0/1796109554 wait complete. 2023-12-13T23:55:23.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.979+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faffc069320 con 0x7faffc071260 2023-12-13T23:55:23.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.979+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f92ec0d9380 con 0x7f92ec054100 2023-12-13T23:55:23.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.979+0000 7faff8ff9640 1 -- 172.21.15.124:0/3438315089 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faff0079e70 con 0x7faffc071260 2023-12-13T23:55:23.978 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:23.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.979+0000 7f92f17fa640 1 -- 172.21.15.124:0/1974052054 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f92e40585c0 con 0x7f92ec054100 2023-12-13T23:55:23.978 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:23.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-12-13T23:55:23.979 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 msgr2=0x7fafe006d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 0x7fafe006d750 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faffc1573f0 tx=0x7fafe8009060 comp rx=0 tx=0).stop 2023-12-13T23:55:23.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 msgr2=0x7faffc14b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7faff00097f0 tx=0x7faff00046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 shutdown_connections 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafe006b290 0x7fafe006d750 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faffc1597f0 0x7faffc14eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faffc156dd0 0x7faffc157e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 --2- 172.21.15.124:0/3438315089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faffc071260 0x7faffc14b410 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.981+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 >> 172.21.15.124:0/3438315089 conn(0x7faffc0b0c60 msgr2=0x7faffc0b1040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 shutdown_connections 2023-12-13T23:55:23.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7fb001575640 1 -- 172.21.15.124:0/3438315089 wait complete. 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 msgr2=0x7f92e006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 0x7f92e006d680 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f92f404f5f0 tx=0x7f92f406d040 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 msgr2=0x7f92ec052890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f92e400d7d0 tx=0x7f92e400dc90 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 shutdown_connections 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f92e006b1c0 0x7f92e006d680 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f92ec0e3090 0x7f92ec0d5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f92ec054100 0x7f92ec052890 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 --2- 172.21.15.124:0/1974052054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f92ec03e810 0x7f92ec0d3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.982+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 >> 172.21.15.124:0/1974052054 conn(0x7f92ec0559b0 msgr2=0x7f92ec039bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.983+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 shutdown_connections 2023-12-13T23:55:23.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.983+0000 7f92f9c59640 1 -- 172.21.15.124:0/1974052054 wait complete. 2023-12-13T23:55:23.985 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.5 2023-12-13T23:55:23.985 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.989+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc03415a0d0 con 0x7fc0341502c0 2023-12-13T23:55:23.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.989+0000 7fc029ffb640 1 -- 172.21.15.124:0/2813836626 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc03415a0d0 con 0x7fc0341502c0 2023-12-13T23:55:23.987 INFO:teuthology.orchestra.run.smithi124.stdout:55834574857 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 msgr2=0x7fc01806d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 0x7fc01806d6d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc030036000 tx=0x7fc030030000 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 msgr2=0x7fc03415a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc02c007e60 tx=0x7fc02c00e4b0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 shutdown_connections 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc01806b210 0x7fc01806d6d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc0341502c0 0x7fc03415a3b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc0341651b0 0x7fc03415acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 --2- 172.21.15.124:0/2813836626 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc0340b0cf0 0x7fc034156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:23.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 >> 172.21.15.124:0/2813836626 conn(0x7fc03406a4d0 msgr2=0x7fc0340cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:23.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 shutdown_connections 2023-12-13T23:55:23.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:23.991+0000 7fc03b2bb640 1 -- 172.21.15.124:0/2813836626 wait complete. 2023-12-13T23:55:23.992 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.2 2023-12-13T23:55:23.993 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:23.994 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.1 2023-12-13T23:55:23.994 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:24.000 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.4 2023-12-13T23:55:24.001 DEBUG:teuthology.parallel:result is None 2023-12-13T23:55:24.001 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-13T23:55:24.001 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:24.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.142+0000 7f09351c9640 1 Processor -- start 2023-12-13T23:55:24.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.142+0000 7f09351c9640 1 -- start start 2023-12-13T23:55:24.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f09351c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f093015d330 0x7f09300b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f09351c9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300b2b10 con 0x7f09300d2910 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f09351c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09300d1590 con 0x7f09300d3900 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f09351c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09300d1870 con 0x7f093015d330 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f092e59c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f09300d2910 0x7f09300d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43674/0 (socket says 172.21.15.124:43674) 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f092e59c640 1 -- 172.21.15.124:0/2795075529 learned_addr learned my addr 172.21.15.124:0/2795075529 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f092ed9d640 1 --2- 172.21.15.124:0/2795075529 >> v2:172.21.15.203:3300/0 conn(0x7f093015d330 0x7f09300b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1838604839 0 0) 0x7f09300b2b10 con 0x7f09300d2910 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.143+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0918003520 con 0x7f09300d2910 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1744349652 0 0) 0x7f0918003520 con 0x7f09300d2910 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 >> v2:172.21.15.203:3300/0 conn(0x7f093015d330 msgr2=0x7f09300b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 --2- 172.21.15.124:0/2795075529 >> v2:172.21.15.203:3300/0 conn(0x7f093015d330 0x7f09300b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 >> v1:172.21.15.203:6789/0 conn(0x7f09300d3900 legacy=0x7f09300b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300d1b50 con 0x7f09300d2910 2023-12-13T23:55:24.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0924001670 con 0x7f09300d2910 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 >> v1:172.21.15.124:6789/0 conn(0x7f09300d2910 legacy=0x7f09300d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0918003970 0x7f0918003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f0918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0918005180 0x7f091800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09300d1870 con 0x7f09180047c0 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091800c120 con 0x7f0918003970 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091800c2e0 con 0x7f0918005180 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f0918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092ed9d640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0918003970 0x7f0918003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092f59e640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0918005180 0x7f091800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 -- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0918005180 msgr2=0x7f091800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0918005180 0x7f091800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 -- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0918003970 msgr2=0x7f0918003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0918003970 0x7f0918003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092e59c640 1 -- 172.21.15.124:0/2795075529 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0918003520 con 0x7f09180047c0 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092f59e640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0918005180 0x7f091800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.144+0000 7f092f59e640 1 -- 172.21.15.124:0/2795075529 reap_dead start 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f092e59c640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f0918004c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f092400c240 tx=0x7f09180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94a34b3086ea696 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924005ad0 con 0x7f09180047c0 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924005c30 con 0x7f09180047c0 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f092dd9b640 1 -- 172.21.15.124:0/2795075529 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092402caf0 con 0x7f09180047c0 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f09351c9640 1 -- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 msgr2=0x7f0918004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f09351c9640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f0918004c40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f092400c240 tx=0x7f09180057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f09351c9640 1 -- 172.21.15.124:0/2795075529 shutdown_connections 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.145+0000 7f09351c9640 1 --2- 172.21.15.124:0/2795075529 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f0918004c40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.146+0000 7f09351c9640 1 -- 172.21.15.124:0/2795075529 >> 172.21.15.124:0/2795075529 conn(0x7f0930072020 msgr2=0x7f0930072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.146+0000 7f09351c9640 1 -- 172.21.15.124:0/2795075529 shutdown_connections 2023-12-13T23:55:24.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.146+0000 7f09351c9640 1 -- 172.21.15.124:0/2795075529 wait complete. 2023-12-13T23:55:24.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.147+0000 7f09351c9640 1 Processor -- start 2023-12-13T23:55:24.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.147+0000 7f09351c9640 1 -- start start 2023-12-13T23:55:24.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09300cca70 0x7f0930160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0930154d90 0x7f0930155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09301f60b0 con 0x7f09180047c0 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0930160980 con 0x7f0930154d90 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09300cb9d0 con 0x7f09300cca70 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57468/0 (socket says 172.21.15.124:57468) 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 -- 172.21.15.124:0/3329631801 learned_addr learned my addr 172.21.15.124:0/3329631801 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092f59e640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0930154d90 0x7f0930155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092e59c640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09300cca70 0x7f0930160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 -- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09300cca70 msgr2=0x7f0930160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09300cca70 0x7f0930160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 -- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0930154d90 msgr2=0x7f0930155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0930154d90 0x7f0930155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 -- 172.21.15.124:0/3329631801 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09300cbcb0 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f092ed9d640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0920011e30 tx=0x7f092000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f092000e040 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f092000f040 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0920009c00 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09300cbf90 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.148+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09300cc550 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.149+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0930069400 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.150+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0920015850 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.150+0000 7f09177fe640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 0x7f091c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.150+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f092008a810 con 0x7f09180047c0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.150+0000 7f092e59c640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 0x7f091c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.151+0000 7f092e59c640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 0x7f091c06d6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0924034000 tx=0x7f0924009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.155+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f09200555d0 con 0x7f09180047c0 2023-12-13T23:55:24.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.319+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f09300d23f0 con 0x7f091c06b1f0 2023-12-13T23:55:24.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.326+0000 7f09177fe640 1 -- 172.21.15.124:0/3329631801 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264619 (secure 0 0 0) 0x7f092403d040 con 0x7f091c06b1f0 2023-12-13T23:55:24.328 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:24.329 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:24.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 msgr2=0x7f091c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 0x7f091c06d6b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0924034000 tx=0x7f0924009030 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 msgr2=0x7f09300cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0920011e30 tx=0x7f092000f580 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 shutdown_connections 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f091c06b1f0 0x7f091c06d6b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0930154d90 0x7f0930155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f09300cca70 0x7f0930160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 --2- 172.21.15.124:0/3329631801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09180047c0 0x7f09300cb380 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 >> 172.21.15.124:0/3329631801 conn(0x7f0930072020 msgr2=0x7f09300b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 shutdown_connections 2023-12-13T23:55:24.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.329+0000 7f09351c9640 1 -- 172.21.15.124:0/3329631801 wait complete. 2023-12-13T23:55:24.338 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-12-13T23:55:24.112195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.373762"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741160+0000","last_change":"2023-12-13T23:55:12.749065+0000","last_active":"2023-12-13T23:55:13.741160+0000","last_peered":"2023-12-13T23:55:13.741160+0000","last_clean":"2023-12-13T23:55:13.741160+0000","last_became_active":"2023-12-13T23:55:12.748399+0000","last_became_peered":"2023-12-13T23:55:12.748399+0000","last_unstale":"2023-12-13T23:55:13.741160+0000","last_undegraded":"2023-12-13T23:55:13.741160+0000","last_fullsized":"2023-12-13T23:55:13.741160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613725+0000","last_change":"2023-12-13T23:55:12.751938+0000","last_active":"2023-12-13T23:55:14.613725+0000","last_peered":"2023-12-13T23:55:14.613725+0000","last_clean":"2023-12-13T23:55:14.613725+0000","last_became_active":"2023-12-13T23:55:12.751688+0000","last_became_peered":"2023-12-13T23:55:12.751688+0000","last_unstale":"2023-12-13T23:55:14.613725+0000","last_undegraded":"2023-12-13T23:55:14.613725+0000","last_fullsized":"2023-12-13T23:55:14.613725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624101+0000","last_change":"2023-12-13T23:55:12.757831+0000","last_active":"2023-12-13T23:55:14.624101+0000","last_peered":"2023-12-13T23:55:14.624101+0000","last_clean":"2023-12-13T23:55:14.624101+0000","last_became_active":"2023-12-13T23:55:12.757403+0000","last_became_peered":"2023-12-13T23:55:12.757403+0000","last_unstale":"2023-12-13T23:55:14.624101+0000","last_undegraded":"2023-12-13T23:55:14.624101+0000","last_fullsized":"2023-12-13T23:55:14.624101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624401+0000","last_change":"2023-12-13T23:55:12.757838+0000","last_active":"2023-12-13T23:55:14.624401+0000","last_peered":"2023-12-13T23:55:14.624401+0000","last_clean":"2023-12-13T23:55:14.624401+0000","last_became_active":"2023-12-13T23:55:12.757590+0000","last_became_peered":"2023-12-13T23:55:12.757590+0000","last_unstale":"2023-12-13T23:55:14.624401+0000","last_undegraded":"2023-12-13T23:55:14.624401+0000","last_fullsized":"2023-12-13T23:55:14.624401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740506+0000","last_change":"2023-12-13T23:55:12.750174+0000","last_active":"2023-12-13T23:55:13.740506+0000","last_peered":"2023-12-13T23:55:13.740506+0000","last_clean":"2023-12-13T23:55:13.740506+0000","last_became_active":"2023-12-13T23:55:12.749937+0000","last_became_peered":"2023-12-13T23:55:12.749937+0000","last_unstale":"2023-12-13T23:55:13.740506+0000","last_undegraded":"2023-12-13T23:55:13.740506+0000","last_fullsized":"2023-12-13T23:55:13.740506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624775+0000","last_change":"2023-12-13T23:55:12.760175+0000","last_active":"2023-12-13T23:55:14.624775+0000","last_peered":"2023-12-13T23:55:14.624775+0000","last_clean":"2023-12-13T23:55:14.624775+0000","last_became_active":"2023-12-13T23:55:12.759888+0000","last_became_peered":"2023-12-13T23:55:12.759888+0000","last_unstale":"2023-12-13T23:55:14.624775+0000","last_undegraded":"2023-12-13T23:55:14.624775+0000","last_fullsized":"2023-12-13T23:55:14.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740922+0000","last_change":"2023-12-13T23:55:12.758545+0000","last_active":"2023-12-13T23:55:13.740922+0000","last_peered":"2023-12-13T23:55:13.740922+0000","last_clean":"2023-12-13T23:55:13.740922+0000","last_became_active":"2023-12-13T23:55:12.758292+0000","last_became_peered":"2023-12-13T23:55:12.758292+0000","last_unstale":"2023-12-13T23:55:13.740922+0000","last_undegraded":"2023-12-13T23:55:13.740922+0000","last_fullsized":"2023-12-13T23:55:13.740922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737487+0000","last_change":"2023-12-13T23:55:12.756050+0000","last_active":"2023-12-13T23:55:13.737487+0000","last_peered":"2023-12-13T23:55:13.737487+0000","last_clean":"2023-12-13T23:55:13.737487+0000","last_became_active":"2023-12-13T23:55:12.755765+0000","last_became_peered":"2023-12-13T23:55:12.755765+0000","last_unstale":"2023-12-13T23:55:13.737487+0000","last_undegraded":"2023-12-13T23:55:13.737487+0000","last_fullsized":"2023-12-13T23:55:13.737487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741293+0000","last_change":"2023-12-13T23:55:12.751022+0000","last_active":"2023-12-13T23:55:13.741293+0000","last_peered":"2023-12-13T23:55:13.741293+0000","last_clean":"2023-12-13T23:55:13.741293+0000","last_became_active":"2023-12-13T23:55:12.750750+0000","last_became_peered":"2023-12-13T23:55:12.750750+0000","last_unstale":"2023-12-13T23:55:13.741293+0000","last_undegraded":"2023-12-13T23:55:13.741293+0000","last_fullsized":"2023-12-13T23:55:13.741293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624534+0000","last_change":"2023-12-13T23:55:12.754201+0000","last_active":"2023-12-13T23:55:14.624534+0000","last_peered":"2023-12-13T23:55:14.624534+0000","last_clean":"2023-12-13T23:55:14.624534+0000","last_became_active":"2023-12-13T23:55:12.753905+0000","last_became_peered":"2023-12-13T23:55:12.753905+0000","last_unstale":"2023-12-13T23:55:14.624534+0000","last_undegraded":"2023-12-13T23:55:14.624534+0000","last_fullsized":"2023-12-13T23:55:14.624534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624445+0000","last_change":"2023-12-13T23:55:12.756766+0000","last_active":"2023-12-13T23:55:14.624445+0000","last_peered":"2023-12-13T23:55:14.624445+0000","last_clean":"2023-12-13T23:55:14.624445+0000","last_became_active":"2023-12-13T23:55:12.756500+0000","last_became_peered":"2023-12-13T23:55:12.756500+0000","last_unstale":"2023-12-13T23:55:14.624445+0000","last_undegraded":"2023-12-13T23:55:14.624445+0000","last_fullsized":"2023-12-13T23:55:14.624445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741255+0000","last_change":"2023-12-13T23:55:12.756454+0000","last_active":"2023-12-13T23:55:13.741255+0000","last_peered":"2023-12-13T23:55:13.741255+0000","last_clean":"2023-12-13T23:55:13.741255+0000","last_became_active":"2023-12-13T23:55:12.756151+0000","last_became_peered":"2023-12-13T23:55:12.756151+0000","last_unstale":"2023-12-13T23:55:13.741255+0000","last_undegraded":"2023-12-13T23:55:13.741255+0000","last_fullsized":"2023-12-13T23:55:13.741255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:41.266746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737767+0000","last_change":"2023-12-13T23:55:12.751798+0000","last_active":"2023-12-13T23:55:13.737767+0000","last_peered":"2023-12-13T23:55:13.737767+0000","last_clean":"2023-12-13T23:55:13.737767+0000","last_became_active":"2023-12-13T23:55:12.751513+0000","last_became_peered":"2023-12-13T23:55:12.751513+0000","last_unstale":"2023-12-13T23:55:13.737767+0000","last_undegraded":"2023-12-13T23:55:13.737767+0000","last_fullsized":"2023-12-13T23:55:13.737767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624265+0000","last_change":"2023-12-13T23:55:12.757612+0000","last_active":"2023-12-13T23:55:14.624265+0000","last_peered":"2023-12-13T23:55:14.624265+0000","last_clean":"2023-12-13T23:55:14.624265+0000","last_became_active":"2023-12-13T23:55:12.757320+0000","last_became_peered":"2023-12-13T23:55:12.757320+0000","last_unstale":"2023-12-13T23:55:14.624265+0000","last_undegraded":"2023-12-13T23:55:14.624265+0000","last_fullsized":"2023-12-13T23:55:14.624265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613523+0000","last_change":"2023-12-13T23:55:12.743417+0000","last_active":"2023-12-13T23:55:14.613523+0000","last_peered":"2023-12-13T23:55:14.613523+0000","last_clean":"2023-12-13T23:55:14.613523+0000","last_became_active":"2023-12-13T23:55:12.743121+0000","last_became_peered":"2023-12-13T23:55:12.743121+0000","last_unstale":"2023-12-13T23:55:14.613523+0000","last_undegraded":"2023-12-13T23:55:14.613523+0000","last_fullsized":"2023-12-13T23:55:14.613523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624106+0000","last_change":"2023-12-13T23:55:12.761124+0000","last_active":"2023-12-13T23:55:14.624106+0000","last_peered":"2023-12-13T23:55:14.624106+0000","last_clean":"2023-12-13T23:55:14.624106+0000","last_became_active":"2023-12-13T23:55:12.760661+0000","last_became_peered":"2023-12-13T23:55:12.760661+0000","last_unstale":"2023-12-13T23:55:14.624106+0000","last_undegraded":"2023-12-13T23:55:14.624106+0000","last_fullsized":"2023-12-13T23:55:14.624106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624905+0000","last_change":"2023-12-13T23:55:12.752470+0000","last_active":"2023-12-13T23:55:14.624905+0000","last_peered":"2023-12-13T23:55:14.624905+0000","last_clean":"2023-12-13T23:55:14.624905+0000","last_became_active":"2023-12-13T23:55:12.752197+0000","last_became_peered":"2023-12-13T23:55:12.752197+0000","last_unstale":"2023-12-13T23:55:14.624905+0000","last_undegraded":"2023-12-13T23:55:14.624905+0000","last_fullsized":"2023-12-13T23:55:14.624905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623990+0000","last_change":"2023-12-13T23:55:12.748844+0000","last_active":"2023-12-13T23:55:14.623990+0000","last_peered":"2023-12-13T23:55:14.623990+0000","last_clean":"2023-12-13T23:55:14.623990+0000","last_became_active":"2023-12-13T23:55:12.748564+0000","last_became_peered":"2023-12-13T23:55:12.748564+0000","last_unstale":"2023-12-13T23:55:14.623990+0000","last_undegraded":"2023-12-13T23:55:14.623990+0000","last_fullsized":"2023-12-13T23:55:14.623990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741142+0000","last_change":"2023-12-13T23:55:12.749584+0000","last_active":"2023-12-13T23:55:13.741142+0000","last_peered":"2023-12-13T23:55:13.741142+0000","last_clean":"2023-12-13T23:55:13.741142+0000","last_became_active":"2023-12-13T23:55:12.749330+0000","last_became_peered":"2023-12-13T23:55:12.749330+0000","last_unstale":"2023-12-13T23:55:13.741142+0000","last_undegraded":"2023-12-13T23:55:13.741142+0000","last_fullsized":"2023-12-13T23:55:13.741142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624156+0000","last_change":"2023-12-13T23:55:12.746390+0000","last_active":"2023-12-13T23:55:14.624156+0000","last_peered":"2023-12-13T23:55:14.624156+0000","last_clean":"2023-12-13T23:55:14.624156+0000","last_became_active":"2023-12-13T23:55:12.746135+0000","last_became_peered":"2023-12-13T23:55:12.746135+0000","last_unstale":"2023-12-13T23:55:14.624156+0000","last_undegraded":"2023-12-13T23:55:14.624156+0000","last_fullsized":"2023-12-13T23:55:14.624156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624034+0000","last_change":"2023-12-13T23:55:12.752805+0000","last_active":"2023-12-13T23:55:14.624034+0000","last_peered":"2023-12-13T23:55:14.624034+0000","last_clean":"2023-12-13T23:55:14.624034+0000","last_became_active":"2023-12-13T23:55:12.752447+0000","last_became_peered":"2023-12-13T23:55:12.752447+0000","last_unstale":"2023-12-13T23:55:14.624034+0000","last_undegraded":"2023-12-13T23:55:14.624034+0000","last_fullsized":"2023-12-13T23:55:14.624034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:24.291337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624917+0000","last_change":"2023-12-13T23:55:12.754499+0000","last_active":"2023-12-13T23:55:14.624917+0000","last_peered":"2023-12-13T23:55:14.624917+0000","last_clean":"2023-12-13T23:55:14.624917+0000","last_became_active":"2023-12-13T23:55:12.753996+0000","last_became_peered":"2023-12-13T23:55:12.753996+0000","last_unstale":"2023-12-13T23:55:14.624917+0000","last_undegraded":"2023-12-13T23:55:14.624917+0000","last_fullsized":"2023-12-13T23:55:14.624917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737534+0000","last_change":"2023-12-13T23:55:12.750598+0000","last_active":"2023-12-13T23:55:13.737534+0000","last_peered":"2023-12-13T23:55:13.737534+0000","last_clean":"2023-12-13T23:55:13.737534+0000","last_became_active":"2023-12-13T23:55:12.750296+0000","last_became_peered":"2023-12-13T23:55:12.750296+0000","last_unstale":"2023-12-13T23:55:13.737534+0000","last_undegraded":"2023-12-13T23:55:13.737534+0000","last_fullsized":"2023-12-13T23:55:13.737534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624327+0000","last_change":"2023-12-13T23:55:12.750435+0000","last_active":"2023-12-13T23:55:14.624327+0000","last_peered":"2023-12-13T23:55:14.624327+0000","last_clean":"2023-12-13T23:55:14.624327+0000","last_became_active":"2023-12-13T23:55:12.750151+0000","last_became_peered":"2023-12-13T23:55:12.750151+0000","last_unstale":"2023-12-13T23:55:14.624327+0000","last_undegraded":"2023-12-13T23:55:14.624327+0000","last_fullsized":"2023-12-13T23:55:14.624327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737900+0000","last_change":"2023-12-13T23:55:12.750495+0000","last_active":"2023-12-13T23:55:13.737900+0000","last_peered":"2023-12-13T23:55:13.737900+0000","last_clean":"2023-12-13T23:55:13.737900+0000","last_became_active":"2023-12-13T23:55:12.750227+0000","last_became_peered":"2023-12-13T23:55:12.750227+0000","last_unstale":"2023-12-13T23:55:13.737900+0000","last_undegraded":"2023-12-13T23:55:13.737900+0000","last_fullsized":"2023-12-13T23:55:13.737900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624667+0000","last_change":"2023-12-13T23:55:12.751349+0000","last_active":"2023-12-13T23:55:14.624667+0000","last_peered":"2023-12-13T23:55:14.624667+0000","last_clean":"2023-12-13T23:55:14.624667+0000","last_became_active":"2023-12-13T23:55:12.751057+0000","last_became_peered":"2023-12-13T23:55:12.751057+0000","last_unstale":"2023-12-13T23:55:14.624667+0000","last_undegraded":"2023-12-13T23:55:14.624667+0000","last_fullsized":"2023-12-13T23:55:14.624667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613647+0000","last_change":"2023-12-13T23:55:12.748079+0000","last_active":"2023-12-13T23:55:14.613647+0000","last_peered":"2023-12-13T23:55:14.613647+0000","last_clean":"2023-12-13T23:55:14.613647+0000","last_became_active":"2023-12-13T23:55:12.747722+0000","last_became_peered":"2023-12-13T23:55:12.747722+0000","last_unstale":"2023-12-13T23:55:14.613647+0000","last_undegraded":"2023-12-13T23:55:14.613647+0000","last_fullsized":"2023-12-13T23:55:14.613647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737802+0000","last_change":"2023-12-13T23:55:12.747671+0000","last_active":"2023-12-13T23:55:13.737802+0000","last_peered":"2023-12-13T23:55:13.737802+0000","last_clean":"2023-12-13T23:55:13.737802+0000","last_became_active":"2023-12-13T23:55:12.747431+0000","last_became_peered":"2023-12-13T23:55:12.747431+0000","last_unstale":"2023-12-13T23:55:13.737802+0000","last_undegraded":"2023-12-13T23:55:13.737802+0000","last_fullsized":"2023-12-13T23:55:13.737802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624186+0000","last_change":"2023-12-13T23:55:12.751994+0000","last_active":"2023-12-13T23:55:14.624186+0000","last_peered":"2023-12-13T23:55:14.624186+0000","last_clean":"2023-12-13T23:55:14.624186+0000","last_became_active":"2023-12-13T23:55:12.751598+0000","last_became_peered":"2023-12-13T23:55:12.751598+0000","last_unstale":"2023-12-13T23:55:14.624186+0000","last_undegraded":"2023-12-13T23:55:14.624186+0000","last_fullsized":"2023-12-13T23:55:14.624186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624938+0000","last_change":"2023-12-13T23:55:12.754312+0000","last_active":"2023-12-13T23:55:14.624938+0000","last_peered":"2023-12-13T23:55:14.624938+0000","last_clean":"2023-12-13T23:55:14.624938+0000","last_became_active":"2023-12-13T23:55:12.753618+0000","last_became_peered":"2023-12-13T23:55:12.753618+0000","last_unstale":"2023-12-13T23:55:14.624938+0000","last_undegraded":"2023-12-13T23:55:14.624938+0000","last_fullsized":"2023-12-13T23:55:14.624938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623822+0000","last_change":"2023-12-13T23:55:12.755004+0000","last_active":"2023-12-13T23:55:14.623822+0000","last_peered":"2023-12-13T23:55:14.623822+0000","last_clean":"2023-12-13T23:55:14.623822+0000","last_became_active":"2023-12-13T23:55:12.754761+0000","last_became_peered":"2023-12-13T23:55:12.754761+0000","last_unstale":"2023-12-13T23:55:14.623822+0000","last_undegraded":"2023-12-13T23:55:14.623822+0000","last_fullsized":"2023-12-13T23:55:14.623822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624213+0000","last_change":"2023-12-13T23:55:12.758828+0000","last_active":"2023-12-13T23:55:14.624213+0000","last_peered":"2023-12-13T23:55:14.624213+0000","last_clean":"2023-12-13T23:55:14.624213+0000","last_became_active":"2023-12-13T23:55:12.758520+0000","last_became_peered":"2023-12-13T23:55:12.758520+0000","last_unstale":"2023-12-13T23:55:14.624213+0000","last_undegraded":"2023-12-13T23:55:14.624213+0000","last_fullsized":"2023-12-13T23:55:14.624213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740370+0000","last_change":"2023-12-13T23:55:12.745882+0000","last_active":"2023-12-13T23:55:13.740370+0000","last_peered":"2023-12-13T23:55:13.740370+0000","last_clean":"2023-12-13T23:55:13.740370+0000","last_became_active":"2023-12-13T23:55:12.744988+0000","last_became_peered":"2023-12-13T23:55:12.744988+0000","last_unstale":"2023-12-13T23:55:13.740370+0000","last_undegraded":"2023-12-13T23:55:13.740370+0000","last_fullsized":"2023-12-13T23:55:13.740370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:57.837875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737851+0000","last_change":"2023-12-13T23:55:11.769713+0000","last_active":"2023-12-13T23:55:13.737851+0000","last_peered":"2023-12-13T23:55:13.737851+0000","last_clean":"2023-12-13T23:55:13.737851+0000","last_became_active":"2023-12-13T23:55:11.769540+0000","last_became_peered":"2023-12-13T23:55:11.769540+0000","last_unstale":"2023-12-13T23:55:13.737851+0000","last_undegraded":"2023-12-13T23:55:13.737851+0000","last_fullsized":"2023-12-13T23:55:13.737851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624177+0000","last_change":"2023-12-13T23:55:12.752916+0000","last_active":"2023-12-13T23:55:14.624177+0000","last_peered":"2023-12-13T23:55:14.624177+0000","last_clean":"2023-12-13T23:55:14.624177+0000","last_became_active":"2023-12-13T23:55:12.751877+0000","last_became_peered":"2023-12-13T23:55:12.751877+0000","last_unstale":"2023-12-13T23:55:14.624177+0000","last_undegraded":"2023-12-13T23:55:14.624177+0000","last_fullsized":"2023-12-13T23:55:14.624177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624064+0000","last_change":"2023-12-13T23:55:11.767168+0000","last_active":"2023-12-13T23:55:14.624064+0000","last_peered":"2023-12-13T23:55:14.624064+0000","last_clean":"2023-12-13T23:55:14.624064+0000","last_became_active":"2023-12-13T23:55:11.766944+0000","last_became_peered":"2023-12-13T23:55:11.766944+0000","last_unstale":"2023-12-13T23:55:14.624064+0000","last_undegraded":"2023-12-13T23:55:14.624064+0000","last_fullsized":"2023-12-13T23:55:14.624064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737587+0000","last_change":"2023-12-13T23:55:11.764302+0000","last_active":"2023-12-13T23:55:13.737587+0000","last_peered":"2023-12-13T23:55:13.737587+0000","last_clean":"2023-12-13T23:55:13.737587+0000","last_became_active":"2023-12-13T23:55:11.763930+0000","last_became_peered":"2023-12-13T23:55:11.763930+0000","last_unstale":"2023-12-13T23:55:13.737587+0000","last_undegraded":"2023-12-13T23:55:13.737587+0000","last_fullsized":"2023-12-13T23:55:13.737587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737560+0000","last_change":"2023-12-13T23:55:12.760219+0000","last_active":"2023-12-13T23:55:13.737560+0000","last_peered":"2023-12-13T23:55:13.737560+0000","last_clean":"2023-12-13T23:55:13.737560+0000","last_became_active":"2023-12-13T23:55:12.759890+0000","last_became_peered":"2023-12-13T23:55:12.759890+0000","last_unstale":"2023-12-13T23:55:13.737560+0000","last_undegraded":"2023-12-13T23:55:13.737560+0000","last_fullsized":"2023-12-13T23:55:13.737560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624404+0000","last_change":"2023-12-13T23:55:11.763128+0000","last_active":"2023-12-13T23:55:14.624404+0000","last_peered":"2023-12-13T23:55:14.624404+0000","last_clean":"2023-12-13T23:55:14.624404+0000","last_became_active":"2023-12-13T23:55:11.762848+0000","last_became_peered":"2023-12-13T23:55:11.762848+0000","last_unstale":"2023-12-13T23:55:14.624404+0000","last_undegraded":"2023-12-13T23:55:14.624404+0000","last_fullsized":"2023-12-13T23:55:14.624404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613510+0000","last_change":"2023-12-13T23:55:12.747367+0000","last_active":"2023-12-13T23:55:14.613510+0000","last_peered":"2023-12-13T23:55:14.613510+0000","last_clean":"2023-12-13T23:55:14.613510+0000","last_became_active":"2023-12-13T23:55:12.747074+0000","last_became_peered":"2023-12-13T23:55:12.747074+0000","last_unstale":"2023-12-13T23:55:14.613510+0000","last_undegraded":"2023-12-13T23:55:14.613510+0000","last_fullsized":"2023-12-13T23:55:14.613510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624348+0000","last_change":"2023-12-13T23:55:11.766140+0000","last_active":"2023-12-13T23:55:14.624348+0000","last_peered":"2023-12-13T23:55:14.624348+0000","last_clean":"2023-12-13T23:55:14.624348+0000","last_became_active":"2023-12-13T23:55:11.765848+0000","last_became_peered":"2023-12-13T23:55:11.765848+0000","last_unstale":"2023-12-13T23:55:14.624348+0000","last_undegraded":"2023-12-13T23:55:14.624348+0000","last_fullsized":"2023-12-13T23:55:14.624348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624786+0000","last_change":"2023-12-13T23:55:12.750678+0000","last_active":"2023-12-13T23:55:14.624786+0000","last_peered":"2023-12-13T23:55:14.624786+0000","last_clean":"2023-12-13T23:55:14.624786+0000","last_became_active":"2023-12-13T23:55:12.750200+0000","last_became_peered":"2023-12-13T23:55:12.750200+0000","last_unstale":"2023-12-13T23:55:14.624786+0000","last_undegraded":"2023-12-13T23:55:14.624786+0000","last_fullsized":"2023-12-13T23:55:14.624786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624863+0000","last_change":"2023-12-13T23:55:11.763844+0000","last_active":"2023-12-13T23:55:14.624863+0000","last_peered":"2023-12-13T23:55:14.624863+0000","last_clean":"2023-12-13T23:55:14.624863+0000","last_became_active":"2023-12-13T23:55:11.763574+0000","last_became_peered":"2023-12-13T23:55:11.763574+0000","last_unstale":"2023-12-13T23:55:14.624863+0000","last_undegraded":"2023-12-13T23:55:14.624863+0000","last_fullsized":"2023-12-13T23:55:14.624863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:41:34.696890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624340+0000","last_change":"2023-12-13T23:55:12.754675+0000","last_active":"2023-12-13T23:55:14.624340+0000","last_peered":"2023-12-13T23:55:14.624340+0000","last_clean":"2023-12-13T23:55:14.624340+0000","last_became_active":"2023-12-13T23:55:12.753740+0000","last_became_peered":"2023-12-13T23:55:12.753740+0000","last_unstale":"2023-12-13T23:55:14.624340+0000","last_undegraded":"2023-12-13T23:55:14.624340+0000","last_fullsized":"2023-12-13T23:55:14.624340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613654+0000","last_change":"2023-12-13T23:55:11.765823+0000","last_active":"2023-12-13T23:55:14.613654+0000","last_peered":"2023-12-13T23:55:14.613654+0000","last_clean":"2023-12-13T23:55:14.613654+0000","last_became_active":"2023-12-13T23:55:11.765515+0000","last_became_peered":"2023-12-13T23:55:11.765515+0000","last_unstale":"2023-12-13T23:55:14.613654+0000","last_undegraded":"2023-12-13T23:55:14.613654+0000","last_fullsized":"2023-12-13T23:55:14.613654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741390+0000","last_change":"2023-12-13T23:55:12.750387+0000","last_active":"2023-12-13T23:55:13.741390+0000","last_peered":"2023-12-13T23:55:13.741390+0000","last_clean":"2023-12-13T23:55:13.741390+0000","last_became_active":"2023-12-13T23:55:12.749101+0000","last_became_peered":"2023-12-13T23:55:12.749101+0000","last_unstale":"2023-12-13T23:55:13.741390+0000","last_undegraded":"2023-12-13T23:55:13.741390+0000","last_fullsized":"2023-12-13T23:55:13.741390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624137+0000","last_change":"2023-12-13T23:55:11.763071+0000","last_active":"2023-12-13T23:55:14.624137+0000","last_peered":"2023-12-13T23:55:14.624137+0000","last_clean":"2023-12-13T23:55:14.624137+0000","last_became_active":"2023-12-13T23:55:11.762774+0000","last_became_peered":"2023-12-13T23:55:11.762774+0000","last_unstale":"2023-12-13T23:55:14.624137+0000","last_undegraded":"2023-12-13T23:55:14.624137+0000","last_fullsized":"2023-12-13T23:55:14.624137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740649+0000","last_change":"2023-12-13T23:55:12.747891+0000","last_active":"2023-12-13T23:55:13.740649+0000","last_peered":"2023-12-13T23:55:13.740649+0000","last_clean":"2023-12-13T23:55:13.740649+0000","last_became_active":"2023-12-13T23:55:12.747629+0000","last_became_peered":"2023-12-13T23:55:12.747629+0000","last_unstale":"2023-12-13T23:55:13.740649+0000","last_undegraded":"2023-12-13T23:55:13.740649+0000","last_fullsized":"2023-12-13T23:55:13.740649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624263+0000","last_change":"2023-12-13T23:55:11.768277+0000","last_active":"2023-12-13T23:55:14.624263+0000","last_peered":"2023-12-13T23:55:14.624263+0000","last_clean":"2023-12-13T23:55:14.624263+0000","last_became_active":"2023-12-13T23:55:11.768039+0000","last_became_peered":"2023-12-13T23:55:11.768039+0000","last_unstale":"2023-12-13T23:55:14.624263+0000","last_undegraded":"2023-12-13T23:55:14.624263+0000","last_fullsized":"2023-12-13T23:55:14.624263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624351+0000","last_change":"2023-12-13T23:55:12.758134+0000","last_active":"2023-12-13T23:55:14.624351+0000","last_peered":"2023-12-13T23:55:14.624351+0000","last_clean":"2023-12-13T23:55:14.624351+0000","last_became_active":"2023-12-13T23:55:12.757679+0000","last_became_peered":"2023-12-13T23:55:12.757679+0000","last_unstale":"2023-12-13T23:55:14.624351+0000","last_undegraded":"2023-12-13T23:55:14.624351+0000","last_fullsized":"2023-12-13T23:55:14.624351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625036+0000","last_change":"2023-12-13T23:55:11.765748+0000","last_active":"2023-12-13T23:55:14.625036+0000","last_peered":"2023-12-13T23:55:14.625036+0000","last_clean":"2023-12-13T23:55:14.625036+0000","last_became_active":"2023-12-13T23:55:11.765320+0000","last_became_peered":"2023-12-13T23:55:11.765320+0000","last_unstale":"2023-12-13T23:55:14.625036+0000","last_undegraded":"2023-12-13T23:55:14.625036+0000","last_fullsized":"2023-12-13T23:55:14.625036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740530+0000","last_change":"2023-12-13T23:55:12.747127+0000","last_active":"2023-12-13T23:55:13.740530+0000","last_peered":"2023-12-13T23:55:13.740530+0000","last_clean":"2023-12-13T23:55:13.740530+0000","last_became_active":"2023-12-13T23:55:12.746838+0000","last_became_peered":"2023-12-13T23:55:12.746838+0000","last_unstale":"2023-12-13T23:55:13.740530+0000","last_undegraded":"2023-12-13T23:55:13.740530+0000","last_fullsized":"2023-12-13T23:55:13.740530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623961+0000","last_change":"2023-12-13T23:55:11.765098+0000","last_active":"2023-12-13T23:55:14.623961+0000","last_peered":"2023-12-13T23:55:14.623961+0000","last_clean":"2023-12-13T23:55:14.623961+0000","last_became_active":"2023-12-13T23:55:11.764746+0000","last_became_peered":"2023-12-13T23:55:11.764746+0000","last_unstale":"2023-12-13T23:55:14.623961+0000","last_undegraded":"2023-12-13T23:55:14.623961+0000","last_fullsized":"2023-12-13T23:55:14.623961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624195+0000","last_change":"2023-12-13T23:55:12.759703+0000","last_active":"2023-12-13T23:55:14.624195+0000","last_peered":"2023-12-13T23:55:14.624195+0000","last_clean":"2023-12-13T23:55:14.624195+0000","last_became_active":"2023-12-13T23:55:12.759371+0000","last_became_peered":"2023-12-13T23:55:12.759371+0000","last_unstale":"2023-12-13T23:55:14.624195+0000","last_undegraded":"2023-12-13T23:55:14.624195+0000","last_fullsized":"2023-12-13T23:55:14.624195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623693+0000","last_change":"2023-12-13T23:55:11.757065+0000","last_active":"2023-12-13T23:55:14.623693+0000","last_peered":"2023-12-13T23:55:14.623693+0000","last_clean":"2023-12-13T23:55:14.623693+0000","last_became_active":"2023-12-13T23:55:11.756732+0000","last_became_peered":"2023-12-13T23:55:11.756732+0000","last_unstale":"2023-12-13T23:55:14.623693+0000","last_undegraded":"2023-12-13T23:55:14.623693+0000","last_fullsized":"2023-12-13T23:55:14.623693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740938+0000","last_change":"2023-12-13T23:55:12.747168+0000","last_active":"2023-12-13T23:55:13.740938+0000","last_peered":"2023-12-13T23:55:13.740938+0000","last_clean":"2023-12-13T23:55:13.740938+0000","last_became_active":"2023-12-13T23:55:12.746860+0000","last_became_peered":"2023-12-13T23:55:12.746860+0000","last_unstale":"2023-12-13T23:55:13.740938+0000","last_undegraded":"2023-12-13T23:55:13.740938+0000","last_fullsized":"2023-12-13T23:55:13.740938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740851+0000","last_change":"2023-12-13T23:55:11.765922+0000","last_active":"2023-12-13T23:55:13.740851+0000","last_peered":"2023-12-13T23:55:13.740851+0000","last_clean":"2023-12-13T23:55:13.740851+0000","last_became_active":"2023-12-13T23:55:11.765641+0000","last_became_peered":"2023-12-13T23:55:11.765641+0000","last_unstale":"2023-12-13T23:55:13.740851+0000","last_undegraded":"2023-12-13T23:55:13.740851+0000","last_fullsized":"2023-12-13T23:55:13.740851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:07:21.569553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741605+0000","last_change":"2023-12-13T23:55:12.747433+0000","last_active":"2023-12-13T23:55:13.741605+0000","last_peered":"2023-12-13T23:55:13.741605+0000","last_clean":"2023-12-13T23:55:13.741605+0000","last_became_active":"2023-12-13T23:55:12.747169+0000","last_became_peered":"2023-12-13T23:55:12.747169+0000","last_unstale":"2023-12-13T23:55:13.741605+0000","last_undegraded":"2023-12-13T23:55:13.741605+0000","last_fullsized":"2023-12-13T23:55:13.741605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:03:57.776819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624072+0000","last_change":"2023-12-13T23:55:11.762985+0000","last_active":"2023-12-13T23:55:14.624072+0000","last_peered":"2023-12-13T23:55:14.624072+0000","last_clean":"2023-12-13T23:55:14.624072+0000","last_became_active":"2023-12-13T23:55:11.762654+0000","last_became_peered":"2023-12-13T23:55:11.762654+0000","last_unstale":"2023-12-13T23:55:14.624072+0000","last_undegraded":"2023-12-13T23:55:14.624072+0000","last_fullsized":"2023-12-13T23:55:14.624072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741322+0000","last_change":"2023-12-13T23:55:12.756265+0000","last_active":"2023-12-13T23:55:13.741322+0000","last_peered":"2023-12-13T23:55:13.741322+0000","last_clean":"2023-12-13T23:55:13.741322+0000","last_became_active":"2023-12-13T23:55:12.756003+0000","last_became_peered":"2023-12-13T23:55:12.756003+0000","last_unstale":"2023-12-13T23:55:13.741322+0000","last_undegraded":"2023-12-13T23:55:13.741322+0000","last_fullsized":"2023-12-13T23:55:13.741322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:26:45.885932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613709+0000","last_change":"2023-12-13T23:55:11.766945+0000","last_active":"2023-12-13T23:55:14.613709+0000","last_peered":"2023-12-13T23:55:14.613709+0000","last_clean":"2023-12-13T23:55:14.613709+0000","last_became_active":"2023-12-13T23:55:11.766671+0000","last_became_peered":"2023-12-13T23:55:11.766671+0000","last_unstale":"2023-12-13T23:55:14.613709+0000","last_undegraded":"2023-12-13T23:55:14.613709+0000","last_fullsized":"2023-12-13T23:55:14.613709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624646+0000","last_change":"2023-12-13T23:55:12.752034+0000","last_active":"2023-12-13T23:55:14.624646+0000","last_peered":"2023-12-13T23:55:14.624646+0000","last_clean":"2023-12-13T23:55:14.624646+0000","last_became_active":"2023-12-13T23:55:12.751764+0000","last_became_peered":"2023-12-13T23:55:12.751764+0000","last_unstale":"2023-12-13T23:55:14.624646+0000","last_undegraded":"2023-12-13T23:55:14.624646+0000","last_fullsized":"2023-12-13T23:55:14.624646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623912+0000","last_change":"2023-12-13T23:55:11.765879+0000","last_active":"2023-12-13T23:55:14.623912+0000","last_peered":"2023-12-13T23:55:14.623912+0000","last_clean":"2023-12-13T23:55:14.623912+0000","last_became_active":"2023-12-13T23:55:11.765607+0000","last_became_peered":"2023-12-13T23:55:11.765607+0000","last_unstale":"2023-12-13T23:55:14.623912+0000","last_undegraded":"2023-12-13T23:55:14.623912+0000","last_fullsized":"2023-12-13T23:55:14.623912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:40.262294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740784+0000","last_change":"2023-12-13T23:55:12.756349+0000","last_active":"2023-12-13T23:55:13.740784+0000","last_peered":"2023-12-13T23:55:13.740784+0000","last_clean":"2023-12-13T23:55:13.740784+0000","last_became_active":"2023-12-13T23:55:12.756033+0000","last_became_peered":"2023-12-13T23:55:12.756033+0000","last_unstale":"2023-12-13T23:55:13.740784+0000","last_undegraded":"2023-12-13T23:55:13.740784+0000","last_fullsized":"2023-12-13T23:55:13.740784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-13T23:55:12.739242+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:12.739242+0000","last_peered":"2023-12-13T23:55:12.739242+0000","last_clean":"2023-12-13T23:55:12.739242+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:12.739242+0000","last_undegraded":"2023-12-13T23:55:12.739242+0000","last_fullsized":"2023-12-13T23:55:12.739242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624508+0000","last_change":"2023-12-13T23:55:11.758971+0000","last_active":"2023-12-13T23:55:14.624508+0000","last_peered":"2023-12-13T23:55:14.624508+0000","last_clean":"2023-12-13T23:55:14.624508+0000","last_became_active":"2023-12-13T23:55:11.758594+0000","last_became_peered":"2023-12-13T23:55:11.758594+0000","last_unstale":"2023-12-13T23:55:14.624508+0000","last_undegraded":"2023-12-13T23:55:14.624508+0000","last_fullsized":"2023-12-13T23:55:14.624508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737699+0000","last_change":"2023-12-13T23:55:12.755959+0000","last_active":"2023-12-13T23:55:13.737699+0000","last_peered":"2023-12-13T23:55:13.737699+0000","last_clean":"2023-12-13T23:55:13.737699+0000","last_became_active":"2023-12-13T23:55:12.755662+0000","last_became_peered":"2023-12-13T23:55:12.755662+0000","last_unstale":"2023-12-13T23:55:13.737699+0000","last_undegraded":"2023-12-13T23:55:13.737699+0000","last_fullsized":"2023-12-13T23:55:13.737699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740316+0000","last_change":"2023-12-13T23:55:11.767937+0000","last_active":"2023-12-13T23:55:13.740316+0000","last_peered":"2023-12-13T23:55:13.740316+0000","last_clean":"2023-12-13T23:55:13.740316+0000","last_became_active":"2023-12-13T23:55:11.767613+0000","last_became_peered":"2023-12-13T23:55:11.767613+0000","last_unstale":"2023-12-13T23:55:13.740316+0000","last_undegraded":"2023-12-13T23:55:13.740316+0000","last_fullsized":"2023-12-13T23:55:13.740316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:53.188642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624112+0000","last_change":"2023-12-13T23:55:12.757755+0000","last_active":"2023-12-13T23:55:14.624112+0000","last_peered":"2023-12-13T23:55:14.624112+0000","last_clean":"2023-12-13T23:55:14.624112+0000","last_became_active":"2023-12-13T23:55:12.757456+0000","last_became_peered":"2023-12-13T23:55:12.757456+0000","last_unstale":"2023-12-13T23:55:14.624112+0000","last_undegraded":"2023-12-13T23:55:14.624112+0000","last_fullsized":"2023-12-13T23:55:14.624112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623940+0000","last_change":"2023-12-13T23:55:11.755829+0000","last_active":"2023-12-13T23:55:14.623940+0000","last_peered":"2023-12-13T23:55:14.623940+0000","last_clean":"2023-12-13T23:55:14.623940+0000","last_became_active":"2023-12-13T23:55:11.755573+0000","last_became_peered":"2023-12-13T23:55:11.755573+0000","last_unstale":"2023-12-13T23:55:14.623940+0000","last_undegraded":"2023-12-13T23:55:14.623940+0000","last_fullsized":"2023-12-13T23:55:14.623940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:56.833424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613504+0000","last_change":"2023-12-13T23:55:12.749732+0000","last_active":"2023-12-13T23:55:14.613504+0000","last_peered":"2023-12-13T23:55:14.613504+0000","last_clean":"2023-12-13T23:55:14.613504+0000","last_became_active":"2023-12-13T23:55:12.749473+0000","last_became_peered":"2023-12-13T23:55:12.749473+0000","last_unstale":"2023-12-13T23:55:14.613504+0000","last_undegraded":"2023-12-13T23:55:14.613504+0000","last_fullsized":"2023-12-13T23:55:14.613504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625041+0000","last_change":"2023-12-13T23:55:11.765970+0000","last_active":"2023-12-13T23:55:14.625041+0000","last_peered":"2023-12-13T23:55:14.625041+0000","last_clean":"2023-12-13T23:55:14.625041+0000","last_became_active":"2023-12-13T23:55:11.765629+0000","last_became_peered":"2023-12-13T23:55:11.765629+0000","last_unstale":"2023-12-13T23:55:14.625041+0000","last_undegraded":"2023-12-13T23:55:14.625041+0000","last_fullsized":"2023-12-13T23:55:14.625041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741379+0000","last_change":"2023-12-13T23:55:12.758873+0000","last_active":"2023-12-13T23:55:13.741379+0000","last_peered":"2023-12-13T23:55:13.741379+0000","last_clean":"2023-12-13T23:55:13.741379+0000","last_became_active":"2023-12-13T23:55:12.758618+0000","last_became_peered":"2023-12-13T23:55:12.758618+0000","last_unstale":"2023-12-13T23:55:13.741379+0000","last_undegraded":"2023-12-13T23:55:13.741379+0000","last_fullsized":"2023-12-13T23:55:13.741379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613597+0000","last_change":"2023-12-13T23:55:11.762944+0000","last_active":"2023-12-13T23:55:14.613597+0000","last_peered":"2023-12-13T23:55:14.613597+0000","last_clean":"2023-12-13T23:55:14.613597+0000","last_became_active":"2023-12-13T23:55:11.762691+0000","last_became_peered":"2023-12-13T23:55:11.762691+0000","last_unstale":"2023-12-13T23:55:14.613597+0000","last_undegraded":"2023-12-13T23:55:14.613597+0000","last_fullsized":"2023-12-13T23:55:14.613597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625017+0000","last_change":"2023-12-13T23:55:12.755585+0000","last_active":"2023-12-13T23:55:14.625017+0000","last_peered":"2023-12-13T23:55:14.625017+0000","last_clean":"2023-12-13T23:55:14.625017+0000","last_became_active":"2023-12-13T23:55:12.755284+0000","last_became_peered":"2023-12-13T23:55:12.755284+0000","last_unstale":"2023-12-13T23:55:14.625017+0000","last_undegraded":"2023-12-13T23:55:14.625017+0000","last_fullsized":"2023-12-13T23:55:14.625017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741015+0000","last_change":"2023-12-13T23:55:11.770291+0000","last_active":"2023-12-13T23:55:13.741015+0000","last_peered":"2023-12-13T23:55:13.741015+0000","last_clean":"2023-12-13T23:55:13.741015+0000","last_became_active":"2023-12-13T23:55:11.769672+0000","last_became_peered":"2023-12-13T23:55:11.769672+0000","last_unstale":"2023-12-13T23:55:13.741015+0000","last_undegraded":"2023-12-13T23:55:13.741015+0000","last_fullsized":"2023-12-13T23:55:13.741015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613615+0000","last_change":"2023-12-13T23:55:12.749467+0000","last_active":"2023-12-13T23:55:14.613615+0000","last_peered":"2023-12-13T23:55:14.613615+0000","last_clean":"2023-12-13T23:55:14.613615+0000","last_became_active":"2023-12-13T23:55:12.749185+0000","last_became_peered":"2023-12-13T23:55:12.749185+0000","last_unstale":"2023-12-13T23:55:14.613615+0000","last_undegraded":"2023-12-13T23:55:14.613615+0000","last_fullsized":"2023-12-13T23:55:14.613615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:15.121343+0000","last_change":"2023-12-13T23:55:11.763643+0000","last_active":"2023-12-13T23:55:15.121343+0000","last_peered":"2023-12-13T23:55:15.121343+0000","last_clean":"2023-12-13T23:55:15.121343+0000","last_became_active":"2023-12-13T23:55:11.763378+0000","last_became_peered":"2023-12-13T23:55:11.763378+0000","last_unstale":"2023-12-13T23:55:15.121343+0000","last_undegraded":"2023-12-13T23:55:15.121343+0000","last_fullsized":"2023-12-13T23:55:15.121343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737984+0000","last_change":"2023-12-13T23:55:12.752448+0000","last_active":"2023-12-13T23:55:13.737984+0000","last_peered":"2023-12-13T23:55:13.737984+0000","last_clean":"2023-12-13T23:55:13.737984+0000","last_became_active":"2023-12-13T23:55:12.752155+0000","last_became_peered":"2023-12-13T23:55:12.752155+0000","last_unstale":"2023-12-13T23:55:13.737984+0000","last_undegraded":"2023-12-13T23:55:13.737984+0000","last_fullsized":"2023-12-13T23:55:13.737984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624790+0000","last_change":"2023-12-13T23:55:11.768156+0000","last_active":"2023-12-13T23:55:14.624790+0000","last_peered":"2023-12-13T23:55:14.624790+0000","last_clean":"2023-12-13T23:55:14.624790+0000","last_became_active":"2023-12-13T23:55:11.767878+0000","last_became_peered":"2023-12-13T23:55:11.767878+0000","last_unstale":"2023-12-13T23:55:14.624790+0000","last_undegraded":"2023-12-13T23:55:14.624790+0000","last_fullsized":"2023-12-13T23:55:14.624790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624226+0000","last_change":"2023-12-13T23:55:12.751906+0000","last_active":"2023-12-13T23:55:14.624226+0000","last_peered":"2023-12-13T23:55:14.624226+0000","last_clean":"2023-12-13T23:55:14.624226+0000","last_became_active":"2023-12-13T23:55:12.751468+0000","last_became_peered":"2023-12-13T23:55:12.751468+0000","last_unstale":"2023-12-13T23:55:14.624226+0000","last_undegraded":"2023-12-13T23:55:14.624226+0000","last_fullsized":"2023-12-13T23:55:14.624226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740225+0000","last_change":"2023-12-13T23:55:11.768417+0000","last_active":"2023-12-13T23:55:13.740225+0000","last_peered":"2023-12-13T23:55:13.740225+0000","last_clean":"2023-12-13T23:55:13.740225+0000","last_became_active":"2023-12-13T23:55:11.768141+0000","last_became_peered":"2023-12-13T23:55:11.768141+0000","last_unstale":"2023-12-13T23:55:13.740225+0000","last_undegraded":"2023-12-13T23:55:13.740225+0000","last_fullsized":"2023-12-13T23:55:13.740225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740516+0000","last_change":"2023-12-13T23:55:12.751650+0000","last_active":"2023-12-13T23:55:13.740516+0000","last_peered":"2023-12-13T23:55:13.740516+0000","last_clean":"2023-12-13T23:55:13.740516+0000","last_became_active":"2023-12-13T23:55:12.749884+0000","last_became_peered":"2023-12-13T23:55:12.749884+0000","last_unstale":"2023-12-13T23:55:13.740516+0000","last_undegraded":"2023-12-13T23:55:13.740516+0000","last_fullsized":"2023-12-13T23:55:13.740516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:12.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613607+0000","last_change":"2023-12-13T23:55:11.761362+0000","last_active":"2023-12-13T23:55:14.613607+0000","last_peered":"2023-12-13T23:55:14.613607+0000","last_clean":"2023-12-13T23:55:14.613607+0000","last_became_active":"2023-12-13T23:55:11.761091+0000","last_became_peered":"2023-12-13T23:55:11.761091+0000","last_unstale":"2023-12-13T23:55:14.613607+0000","last_undegraded":"2023-12-13T23:55:14.613607+0000","last_fullsized":"2023-12-13T23:55:14.613607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:45.292370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737621+0000","last_change":"2023-12-13T23:55:12.747019+0000","last_active":"2023-12-13T23:55:13.737621+0000","last_peered":"2023-12-13T23:55:13.737621+0000","last_clean":"2023-12-13T23:55:13.737621+0000","last_became_active":"2023-12-13T23:55:12.746720+0000","last_became_peered":"2023-12-13T23:55:12.746720+0000","last_unstale":"2023-12-13T23:55:13.737621+0000","last_undegraded":"2023-12-13T23:55:13.737621+0000","last_fullsized":"2023-12-13T23:55:13.737621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613672+0000","last_change":"2023-12-13T23:55:11.761235+0000","last_active":"2023-12-13T23:55:14.613672+0000","last_peered":"2023-12-13T23:55:14.613672+0000","last_clean":"2023-12-13T23:55:14.613672+0000","last_became_active":"2023-12-13T23:55:11.760961+0000","last_became_peered":"2023-12-13T23:55:11.760961+0000","last_unstale":"2023-12-13T23:55:14.613672+0000","last_undegraded":"2023-12-13T23:55:14.613672+0000","last_fullsized":"2023-12-13T23:55:14.613672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737873+0000","last_change":"2023-12-13T23:55:12.757945+0000","last_active":"2023-12-13T23:55:13.737873+0000","last_peered":"2023-12-13T23:55:13.737873+0000","last_clean":"2023-12-13T23:55:13.737873+0000","last_became_active":"2023-12-13T23:55:12.757685+0000","last_became_peered":"2023-12-13T23:55:12.757685+0000","last_unstale":"2023-12-13T23:55:13.737873+0000","last_undegraded":"2023-12-13T23:55:13.737873+0000","last_fullsized":"2023-12-13T23:55:13.737873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740608+0000","last_change":"2023-12-13T23:55:11.769857+0000","last_active":"2023-12-13T23:55:13.740608+0000","last_peered":"2023-12-13T23:55:13.740608+0000","last_clean":"2023-12-13T23:55:13.740608+0000","last_became_active":"2023-12-13T23:55:11.769503+0000","last_became_peered":"2023-12-13T23:55:11.769503+0000","last_unstale":"2023-12-13T23:55:13.740608+0000","last_undegraded":"2023-12-13T23:55:13.740608+0000","last_fullsized":"2023-12-13T23:55:13.740608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624730+0000","last_change":"2023-12-13T23:55:12.747991+0000","last_active":"2023-12-13T23:55:14.624730+0000","last_peered":"2023-12-13T23:55:14.624730+0000","last_clean":"2023-12-13T23:55:14.624730+0000","last_became_active":"2023-12-13T23:55:12.746787+0000","last_became_peered":"2023-12-13T23:55:12.746787+0000","last_unstale":"2023-12-13T23:55:14.624730+0000","last_undegraded":"2023-12-13T23:55:14.624730+0000","last_fullsized":"2023-12-13T23:55:14.624730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624966+0000","last_change":"2023-12-13T23:55:12.755883+0000","last_active":"2023-12-13T23:55:14.624966+0000","last_peered":"2023-12-13T23:55:14.624966+0000","last_clean":"2023-12-13T23:55:14.624966+0000","last_became_active":"2023-12-13T23:55:12.755573+0000","last_became_peered":"2023-12-13T23:55:12.755573+0000","last_unstale":"2023-12-13T23:55:14.624966+0000","last_undegraded":"2023-12-13T23:55:14.624966+0000","last_fullsized":"2023-12-13T23:55:14.624966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741033+0000","last_change":"2023-12-13T23:55:11.769532+0000","last_active":"2023-12-13T23:55:13.741033+0000","last_peered":"2023-12-13T23:55:13.741033+0000","last_clean":"2023-12-13T23:55:13.741033+0000","last_became_active":"2023-12-13T23:55:11.769251+0000","last_became_peered":"2023-12-13T23:55:11.769251+0000","last_unstale":"2023-12-13T23:55:13.741033+0000","last_undegraded":"2023-12-13T23:55:13.741033+0000","last_fullsized":"2023-12-13T23:55:13.741033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624446+0000","last_change":"2023-12-13T23:55:12.757891+0000","last_active":"2023-12-13T23:55:14.624446+0000","last_peered":"2023-12-13T23:55:14.624446+0000","last_clean":"2023-12-13T23:55:14.624446+0000","last_became_active":"2023-12-13T23:55:12.757643+0000","last_became_peered":"2023-12-13T23:55:12.757643+0000","last_unstale":"2023-12-13T23:55:14.624446+0000","last_undegraded":"2023-12-13T23:55:14.624446+0000","last_fullsized":"2023-12-13T23:55:14.624446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740911+0000","last_change":"2023-12-13T23:55:11.769811+0000","last_active":"2023-12-13T23:55:13.740911+0000","last_peered":"2023-12-13T23:55:13.740911+0000","last_clean":"2023-12-13T23:55:13.740911+0000","last_became_active":"2023-12-13T23:55:11.769483+0000","last_became_peered":"2023-12-13T23:55:11.769483+0000","last_unstale":"2023-12-13T23:55:13.740911+0000","last_undegraded":"2023-12-13T23:55:13.740911+0000","last_fullsized":"2023-12-13T23:55:13.740911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624275+0000","last_change":"2023-12-13T23:55:12.758511+0000","last_active":"2023-12-13T23:55:14.624275+0000","last_peered":"2023-12-13T23:55:14.624275+0000","last_clean":"2023-12-13T23:55:14.624275+0000","last_became_active":"2023-12-13T23:55:12.757824+0000","last_became_peered":"2023-12-13T23:55:12.757824+0000","last_unstale":"2023-12-13T23:55:14.624275+0000","last_undegraded":"2023-12-13T23:55:14.624275+0000","last_fullsized":"2023-12-13T23:55:14.624275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737890+0000","last_change":"2023-12-13T23:55:11.761808+0000","last_active":"2023-12-13T23:55:13.737890+0000","last_peered":"2023-12-13T23:55:13.737890+0000","last_clean":"2023-12-13T23:55:13.737890+0000","last_became_active":"2023-12-13T23:55:11.761559+0000","last_became_peered":"2023-12-13T23:55:11.761559+0000","last_unstale":"2023-12-13T23:55:13.737890+0000","last_undegraded":"2023-12-13T23:55:13.737890+0000","last_fullsized":"2023-12-13T23:55:13.737890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737814+0000","last_change":"2023-12-13T23:55:12.748462+0000","last_active":"2023-12-13T23:55:13.737814+0000","last_peered":"2023-12-13T23:55:13.737814+0000","last_clean":"2023-12-13T23:55:13.737814+0000","last_became_active":"2023-12-13T23:55:12.748166+0000","last_became_peered":"2023-12-13T23:55:12.748166+0000","last_unstale":"2023-12-13T23:55:13.737814+0000","last_undegraded":"2023-12-13T23:55:13.737814+0000","last_fullsized":"2023-12-13T23:55:13.737814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624331+0000","last_change":"2023-12-13T23:55:11.771111+0000","last_active":"2023-12-13T23:55:14.624331+0000","last_peered":"2023-12-13T23:55:14.624331+0000","last_clean":"2023-12-13T23:55:14.624331+0000","last_became_active":"2023-12-13T23:55:11.770795+0000","last_became_peered":"2023-12-13T23:55:11.770795+0000","last_unstale":"2023-12-13T23:55:14.624331+0000","last_undegraded":"2023-12-13T23:55:14.624331+0000","last_fullsized":"2023-12-13T23:55:14.624331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:24.343 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:55:24.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.480+0000 7f588eada640 1 Processor -- start 2023-12-13T23:55:24.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.480+0000 7f588eada640 1 -- start start 2023-12-13T23:55:24.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f588806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f58880ad640 0x7f588806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 0x7f5888160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58880adae0 con 0x7f58880ad640 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58880addc0 con 0x7f58880aeb80 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588eada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58880ae0a0 con 0x7f588815d1d0 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f58880ad640 0x7f588806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f58880ad640 0x7f588806ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57480/0 (socket says 172.21.15.124:57480) 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 -- 172.21.15.124:0/3397235907 learned_addr learned my addr 172.21.15.124:0/3397235907 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588d077640 1 --2- 172.21.15.124:0/3397235907 >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 0x7f5888160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588c876640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f588806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 -- 172.21.15.124:0/3397235907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 msgr2=0x7f588806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f588806b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 -- 172.21.15.124:0/3397235907 >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 msgr2=0x7f5888160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 --2- 172.21.15.124:0/3397235907 >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 0x7f5888160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f5887fff640 1 -- 172.21.15.124:0/3397235907 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58880ae380 con 0x7f58880ad640 2023-12-13T23:55:24.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.481+0000 7f588d077640 1 --2- 172.21.15.124:0/3397235907 >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 0x7f5888160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:24.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f5887fff640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880ad640 0x7f588806ba20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f588806f410 tx=0x7f587002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5978c8e7abba2d70 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f58877fe640 1 -- 172.21.15.124:0/3397235907 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870031870 con 0x7f58880ad640 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f58877fe640 1 -- 172.21.15.124:0/3397235907 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58700319d0 con 0x7f58880ad640 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f58877fe640 1 -- 172.21.15.124:0/3397235907 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870031d60 con 0x7f58880ad640 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f588eada640 1 -- 172.21.15.124:0/3397235907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880ad640 msgr2=0x7f588806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.482+0000 7f588eada640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880ad640 0x7f588806ba20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f588806f410 tx=0x7f587002b910 comp rx=0 tx=0).stop 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 -- 172.21.15.124:0/3397235907 shutdown_connections 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880ad640 0x7f588806ba20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 --2- 172.21.15.124:0/3397235907 >> v2:172.21.15.203:3300/0 conn(0x7f58880aeb80 0x7f5888160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 --2- 172.21.15.124:0/3397235907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f588806b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 -- 172.21.15.124:0/3397235907 >> 172.21.15.124:0/3397235907 conn(0x7f5888070e70 msgr2=0x7f5888071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 -- 172.21.15.124:0/3397235907 shutdown_connections 2023-12-13T23:55:24.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.483+0000 7f588eada640 1 -- 172.21.15.124:0/3397235907 wait complete. 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.484+0000 7f588eada640 1 Processor -- start 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.484+0000 7f588eada640 1 -- start start 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.484+0000 7f588eada640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58880ad640 0x7f5888148b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f58880cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5888149060 con 0x7f58880aeb80 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5888142ec0 con 0x7f58880ad640 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f588815c6f0 con 0x7f588815d1d0 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57490/0 (socket says 172.21.15.124:57490) 2023-12-13T23:55:24.486 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 -- 172.21.15.124:0/2225712963 learned_addr learned my addr 172.21.15.124:0/2225712963 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588c876640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58880ad640 0x7f5888148b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588d077640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f58880cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 -- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 msgr2=0x7f58880cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f58880cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 -- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58880ad640 msgr2=0x7f5888148b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58880ad640 0x7f5888148b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 -- 172.21.15.124:0/2225712963 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58700094d0 con 0x7f58880aeb80 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f5887fff640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f587002b950 tx=0x7f58700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f587003abe0 con 0x7f58880aeb80 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f587003ad60 con 0x7f58880aeb80 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588815c9d0 con 0x7f58880aeb80 2023-12-13T23:55:24.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870044630 con 0x7f58880aeb80 2023-12-13T23:55:24.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.485+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58881434f0 con 0x7f58880aeb80 2023-12-13T23:55:24.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.486+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5888069400 con 0x7f58880aeb80 2023-12-13T23:55:24.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.487+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f587004d450 con 0x7f58880aeb80 2023-12-13T23:55:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.487+0000 7f58857fa640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 0x7f586806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.487+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f587002d040 con 0x7f58880aeb80 2023-12-13T23:55:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.487+0000 7f588c876640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 0x7f586806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.488+0000 7f588c876640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 0x7f586806d6b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5878009aa0 tx=0x7f5878009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.491+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f58700b1020 con 0x7f58880aeb80 2023-12-13T23:55:24.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.652+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f58880b2b10 con 0x7f586806b1f0 2023-12-13T23:55:24.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.659+0000 7f58857fa640 1 -- 172.21.15.124:0/2225712963 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264619 (secure 0 0 0) 0x7f5878011a00 con 0x7f586806b1f0 2023-12-13T23:55:24.661 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:24.662 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 msgr2=0x7f586806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 0x7f586806d6b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5878009aa0 tx=0x7f5878009190 comp rx=0 tx=0).stop 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 msgr2=0x7f58880cd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f587002b950 tx=0x7f58700046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 shutdown_connections 2023-12-13T23:55:24.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f586806b1f0 0x7f586806d6b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f588815d1d0 0x7f58880cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f58880aeb80 0x7f58880cd540 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 --2- 172.21.15.124:0/2225712963 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f58880ad640 0x7f5888148b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 >> 172.21.15.124:0/2225712963 conn(0x7f5888070e70 msgr2=0x7f58880af780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 shutdown_connections 2023-12-13T23:55:24.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.662+0000 7f588eada640 1 -- 172.21.15.124:0/2225712963 wait complete. 2023-12-13T23:55:24.671 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-12-13T23:55:24.112195+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219108,"kb_used_data":2920,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755612,"statfs":{"total":773094113280,"available":772869746688,"internally_reserved":0,"allocated":2990080,"data_stored":1790204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.373762"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741160+0000","last_change":"2023-12-13T23:55:12.749065+0000","last_active":"2023-12-13T23:55:13.741160+0000","last_peered":"2023-12-13T23:55:13.741160+0000","last_clean":"2023-12-13T23:55:13.741160+0000","last_became_active":"2023-12-13T23:55:12.748399+0000","last_became_peered":"2023-12-13T23:55:12.748399+0000","last_unstale":"2023-12-13T23:55:13.741160+0000","last_undegraded":"2023-12-13T23:55:13.741160+0000","last_fullsized":"2023-12-13T23:55:13.741160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613725+0000","last_change":"2023-12-13T23:55:12.751938+0000","last_active":"2023-12-13T23:55:14.613725+0000","last_peered":"2023-12-13T23:55:14.613725+0000","last_clean":"2023-12-13T23:55:14.613725+0000","last_became_active":"2023-12-13T23:55:12.751688+0000","last_became_peered":"2023-12-13T23:55:12.751688+0000","last_unstale":"2023-12-13T23:55:14.613725+0000","last_undegraded":"2023-12-13T23:55:14.613725+0000","last_fullsized":"2023-12-13T23:55:14.613725+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624101+0000","last_change":"2023-12-13T23:55:12.757831+0000","last_active":"2023-12-13T23:55:14.624101+0000","last_peered":"2023-12-13T23:55:14.624101+0000","last_clean":"2023-12-13T23:55:14.624101+0000","last_became_active":"2023-12-13T23:55:12.757403+0000","last_became_peered":"2023-12-13T23:55:12.757403+0000","last_unstale":"2023-12-13T23:55:14.624101+0000","last_undegraded":"2023-12-13T23:55:14.624101+0000","last_fullsized":"2023-12-13T23:55:14.624101+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624401+0000","last_change":"2023-12-13T23:55:12.757838+0000","last_active":"2023-12-13T23:55:14.624401+0000","last_peered":"2023-12-13T23:55:14.624401+0000","last_clean":"2023-12-13T23:55:14.624401+0000","last_became_active":"2023-12-13T23:55:12.757590+0000","last_became_peered":"2023-12-13T23:55:12.757590+0000","last_unstale":"2023-12-13T23:55:14.624401+0000","last_undegraded":"2023-12-13T23:55:14.624401+0000","last_fullsized":"2023-12-13T23:55:14.624401+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740506+0000","last_change":"2023-12-13T23:55:12.750174+0000","last_active":"2023-12-13T23:55:13.740506+0000","last_peered":"2023-12-13T23:55:13.740506+0000","last_clean":"2023-12-13T23:55:13.740506+0000","last_became_active":"2023-12-13T23:55:12.749937+0000","last_became_peered":"2023-12-13T23:55:12.749937+0000","last_unstale":"2023-12-13T23:55:13.740506+0000","last_undegraded":"2023-12-13T23:55:13.740506+0000","last_fullsized":"2023-12-13T23:55:13.740506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624775+0000","last_change":"2023-12-13T23:55:12.760175+0000","last_active":"2023-12-13T23:55:14.624775+0000","last_peered":"2023-12-13T23:55:14.624775+0000","last_clean":"2023-12-13T23:55:14.624775+0000","last_became_active":"2023-12-13T23:55:12.759888+0000","last_became_peered":"2023-12-13T23:55:12.759888+0000","last_unstale":"2023-12-13T23:55:14.624775+0000","last_undegraded":"2023-12-13T23:55:14.624775+0000","last_fullsized":"2023-12-13T23:55:14.624775+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740922+0000","last_change":"2023-12-13T23:55:12.758545+0000","last_active":"2023-12-13T23:55:13.740922+0000","last_peered":"2023-12-13T23:55:13.740922+0000","last_clean":"2023-12-13T23:55:13.740922+0000","last_became_active":"2023-12-13T23:55:12.758292+0000","last_became_peered":"2023-12-13T23:55:12.758292+0000","last_unstale":"2023-12-13T23:55:13.740922+0000","last_undegraded":"2023-12-13T23:55:13.740922+0000","last_fullsized":"2023-12-13T23:55:13.740922+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737487+0000","last_change":"2023-12-13T23:55:12.756050+0000","last_active":"2023-12-13T23:55:13.737487+0000","last_peered":"2023-12-13T23:55:13.737487+0000","last_clean":"2023-12-13T23:55:13.737487+0000","last_became_active":"2023-12-13T23:55:12.755765+0000","last_became_peered":"2023-12-13T23:55:12.755765+0000","last_unstale":"2023-12-13T23:55:13.737487+0000","last_undegraded":"2023-12-13T23:55:13.737487+0000","last_fullsized":"2023-12-13T23:55:13.737487+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:24.387370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741293+0000","last_change":"2023-12-13T23:55:12.751022+0000","last_active":"2023-12-13T23:55:13.741293+0000","last_peered":"2023-12-13T23:55:13.741293+0000","last_clean":"2023-12-13T23:55:13.741293+0000","last_became_active":"2023-12-13T23:55:12.750750+0000","last_became_peered":"2023-12-13T23:55:12.750750+0000","last_unstale":"2023-12-13T23:55:13.741293+0000","last_undegraded":"2023-12-13T23:55:13.741293+0000","last_fullsized":"2023-12-13T23:55:13.741293+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624534+0000","last_change":"2023-12-13T23:55:12.754201+0000","last_active":"2023-12-13T23:55:14.624534+0000","last_peered":"2023-12-13T23:55:14.624534+0000","last_clean":"2023-12-13T23:55:14.624534+0000","last_became_active":"2023-12-13T23:55:12.753905+0000","last_became_peered":"2023-12-13T23:55:12.753905+0000","last_unstale":"2023-12-13T23:55:14.624534+0000","last_undegraded":"2023-12-13T23:55:14.624534+0000","last_fullsized":"2023-12-13T23:55:14.624534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624445+0000","last_change":"2023-12-13T23:55:12.756766+0000","last_active":"2023-12-13T23:55:14.624445+0000","last_peered":"2023-12-13T23:55:14.624445+0000","last_clean":"2023-12-13T23:55:14.624445+0000","last_became_active":"2023-12-13T23:55:12.756500+0000","last_became_peered":"2023-12-13T23:55:12.756500+0000","last_unstale":"2023-12-13T23:55:14.624445+0000","last_undegraded":"2023-12-13T23:55:14.624445+0000","last_fullsized":"2023-12-13T23:55:14.624445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741255+0000","last_change":"2023-12-13T23:55:12.756454+0000","last_active":"2023-12-13T23:55:13.741255+0000","last_peered":"2023-12-13T23:55:13.741255+0000","last_clean":"2023-12-13T23:55:13.741255+0000","last_became_active":"2023-12-13T23:55:12.756151+0000","last_became_peered":"2023-12-13T23:55:12.756151+0000","last_unstale":"2023-12-13T23:55:13.741255+0000","last_undegraded":"2023-12-13T23:55:13.741255+0000","last_fullsized":"2023-12-13T23:55:13.741255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:41.266746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737767+0000","last_change":"2023-12-13T23:55:12.751798+0000","last_active":"2023-12-13T23:55:13.737767+0000","last_peered":"2023-12-13T23:55:13.737767+0000","last_clean":"2023-12-13T23:55:13.737767+0000","last_became_active":"2023-12-13T23:55:12.751513+0000","last_became_peered":"2023-12-13T23:55:12.751513+0000","last_unstale":"2023-12-13T23:55:13.737767+0000","last_undegraded":"2023-12-13T23:55:13.737767+0000","last_fullsized":"2023-12-13T23:55:13.737767+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624265+0000","last_change":"2023-12-13T23:55:12.757612+0000","last_active":"2023-12-13T23:55:14.624265+0000","last_peered":"2023-12-13T23:55:14.624265+0000","last_clean":"2023-12-13T23:55:14.624265+0000","last_became_active":"2023-12-13T23:55:12.757320+0000","last_became_peered":"2023-12-13T23:55:12.757320+0000","last_unstale":"2023-12-13T23:55:14.624265+0000","last_undegraded":"2023-12-13T23:55:14.624265+0000","last_fullsized":"2023-12-13T23:55:14.624265+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613523+0000","last_change":"2023-12-13T23:55:12.743417+0000","last_active":"2023-12-13T23:55:14.613523+0000","last_peered":"2023-12-13T23:55:14.613523+0000","last_clean":"2023-12-13T23:55:14.613523+0000","last_became_active":"2023-12-13T23:55:12.743121+0000","last_became_peered":"2023-12-13T23:55:12.743121+0000","last_unstale":"2023-12-13T23:55:14.613523+0000","last_undegraded":"2023-12-13T23:55:14.613523+0000","last_fullsized":"2023-12-13T23:55:14.613523+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624106+0000","last_change":"2023-12-13T23:55:12.761124+0000","last_active":"2023-12-13T23:55:14.624106+0000","last_peered":"2023-12-13T23:55:14.624106+0000","last_clean":"2023-12-13T23:55:14.624106+0000","last_became_active":"2023-12-13T23:55:12.760661+0000","last_became_peered":"2023-12-13T23:55:12.760661+0000","last_unstale":"2023-12-13T23:55:14.624106+0000","last_undegraded":"2023-12-13T23:55:14.624106+0000","last_fullsized":"2023-12-13T23:55:14.624106+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624905+0000","last_change":"2023-12-13T23:55:12.752470+0000","last_active":"2023-12-13T23:55:14.624905+0000","last_peered":"2023-12-13T23:55:14.624905+0000","last_clean":"2023-12-13T23:55:14.624905+0000","last_became_active":"2023-12-13T23:55:12.752197+0000","last_became_peered":"2023-12-13T23:55:12.752197+0000","last_unstale":"2023-12-13T23:55:14.624905+0000","last_undegraded":"2023-12-13T23:55:14.624905+0000","last_fullsized":"2023-12-13T23:55:14.624905+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:40:12.007669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623990+0000","last_change":"2023-12-13T23:55:12.748844+0000","last_active":"2023-12-13T23:55:14.623990+0000","last_peered":"2023-12-13T23:55:14.623990+0000","last_clean":"2023-12-13T23:55:14.623990+0000","last_became_active":"2023-12-13T23:55:12.748564+0000","last_became_peered":"2023-12-13T23:55:12.748564+0000","last_unstale":"2023-12-13T23:55:14.623990+0000","last_undegraded":"2023-12-13T23:55:14.623990+0000","last_fullsized":"2023-12-13T23:55:14.623990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741142+0000","last_change":"2023-12-13T23:55:12.749584+0000","last_active":"2023-12-13T23:55:13.741142+0000","last_peered":"2023-12-13T23:55:13.741142+0000","last_clean":"2023-12-13T23:55:13.741142+0000","last_became_active":"2023-12-13T23:55:12.749330+0000","last_became_peered":"2023-12-13T23:55:12.749330+0000","last_unstale":"2023-12-13T23:55:13.741142+0000","last_undegraded":"2023-12-13T23:55:13.741142+0000","last_fullsized":"2023-12-13T23:55:13.741142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624156+0000","last_change":"2023-12-13T23:55:12.746390+0000","last_active":"2023-12-13T23:55:14.624156+0000","last_peered":"2023-12-13T23:55:14.624156+0000","last_clean":"2023-12-13T23:55:14.624156+0000","last_became_active":"2023-12-13T23:55:12.746135+0000","last_became_peered":"2023-12-13T23:55:12.746135+0000","last_unstale":"2023-12-13T23:55:14.624156+0000","last_undegraded":"2023-12-13T23:55:14.624156+0000","last_fullsized":"2023-12-13T23:55:14.624156+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624034+0000","last_change":"2023-12-13T23:55:12.752805+0000","last_active":"2023-12-13T23:55:14.624034+0000","last_peered":"2023-12-13T23:55:14.624034+0000","last_clean":"2023-12-13T23:55:14.624034+0000","last_became_active":"2023-12-13T23:55:12.752447+0000","last_became_peered":"2023-12-13T23:55:12.752447+0000","last_unstale":"2023-12-13T23:55:14.624034+0000","last_undegraded":"2023-12-13T23:55:14.624034+0000","last_fullsized":"2023-12-13T23:55:14.624034+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:24.291337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624917+0000","last_change":"2023-12-13T23:55:12.754499+0000","last_active":"2023-12-13T23:55:14.624917+0000","last_peered":"2023-12-13T23:55:14.624917+0000","last_clean":"2023-12-13T23:55:14.624917+0000","last_became_active":"2023-12-13T23:55:12.753996+0000","last_became_peered":"2023-12-13T23:55:12.753996+0000","last_unstale":"2023-12-13T23:55:14.624917+0000","last_undegraded":"2023-12-13T23:55:14.624917+0000","last_fullsized":"2023-12-13T23:55:14.624917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737534+0000","last_change":"2023-12-13T23:55:12.750598+0000","last_active":"2023-12-13T23:55:13.737534+0000","last_peered":"2023-12-13T23:55:13.737534+0000","last_clean":"2023-12-13T23:55:13.737534+0000","last_became_active":"2023-12-13T23:55:12.750296+0000","last_became_peered":"2023-12-13T23:55:12.750296+0000","last_unstale":"2023-12-13T23:55:13.737534+0000","last_undegraded":"2023-12-13T23:55:13.737534+0000","last_fullsized":"2023-12-13T23:55:13.737534+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624327+0000","last_change":"2023-12-13T23:55:12.750435+0000","last_active":"2023-12-13T23:55:14.624327+0000","last_peered":"2023-12-13T23:55:14.624327+0000","last_clean":"2023-12-13T23:55:14.624327+0000","last_became_active":"2023-12-13T23:55:12.750151+0000","last_became_peered":"2023-12-13T23:55:12.750151+0000","last_unstale":"2023-12-13T23:55:14.624327+0000","last_undegraded":"2023-12-13T23:55:14.624327+0000","last_fullsized":"2023-12-13T23:55:14.624327+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737900+0000","last_change":"2023-12-13T23:55:12.750495+0000","last_active":"2023-12-13T23:55:13.737900+0000","last_peered":"2023-12-13T23:55:13.737900+0000","last_clean":"2023-12-13T23:55:13.737900+0000","last_became_active":"2023-12-13T23:55:12.750227+0000","last_became_peered":"2023-12-13T23:55:12.750227+0000","last_unstale":"2023-12-13T23:55:13.737900+0000","last_undegraded":"2023-12-13T23:55:13.737900+0000","last_fullsized":"2023-12-13T23:55:13.737900+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624667+0000","last_change":"2023-12-13T23:55:12.751349+0000","last_active":"2023-12-13T23:55:14.624667+0000","last_peered":"2023-12-13T23:55:14.624667+0000","last_clean":"2023-12-13T23:55:14.624667+0000","last_became_active":"2023-12-13T23:55:12.751057+0000","last_became_peered":"2023-12-13T23:55:12.751057+0000","last_unstale":"2023-12-13T23:55:14.624667+0000","last_undegraded":"2023-12-13T23:55:14.624667+0000","last_fullsized":"2023-12-13T23:55:14.624667+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613647+0000","last_change":"2023-12-13T23:55:12.748079+0000","last_active":"2023-12-13T23:55:14.613647+0000","last_peered":"2023-12-13T23:55:14.613647+0000","last_clean":"2023-12-13T23:55:14.613647+0000","last_became_active":"2023-12-13T23:55:12.747722+0000","last_became_peered":"2023-12-13T23:55:12.747722+0000","last_unstale":"2023-12-13T23:55:14.613647+0000","last_undegraded":"2023-12-13T23:55:14.613647+0000","last_fullsized":"2023-12-13T23:55:14.613647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737802+0000","last_change":"2023-12-13T23:55:12.747671+0000","last_active":"2023-12-13T23:55:13.737802+0000","last_peered":"2023-12-13T23:55:13.737802+0000","last_clean":"2023-12-13T23:55:13.737802+0000","last_became_active":"2023-12-13T23:55:12.747431+0000","last_became_peered":"2023-12-13T23:55:12.747431+0000","last_unstale":"2023-12-13T23:55:13.737802+0000","last_undegraded":"2023-12-13T23:55:13.737802+0000","last_fullsized":"2023-12-13T23:55:13.737802+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624186+0000","last_change":"2023-12-13T23:55:12.751994+0000","last_active":"2023-12-13T23:55:14.624186+0000","last_peered":"2023-12-13T23:55:14.624186+0000","last_clean":"2023-12-13T23:55:14.624186+0000","last_became_active":"2023-12-13T23:55:12.751598+0000","last_became_peered":"2023-12-13T23:55:12.751598+0000","last_unstale":"2023-12-13T23:55:14.624186+0000","last_undegraded":"2023-12-13T23:55:14.624186+0000","last_fullsized":"2023-12-13T23:55:14.624186+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624938+0000","last_change":"2023-12-13T23:55:12.754312+0000","last_active":"2023-12-13T23:55:14.624938+0000","last_peered":"2023-12-13T23:55:14.624938+0000","last_clean":"2023-12-13T23:55:14.624938+0000","last_became_active":"2023-12-13T23:55:12.753618+0000","last_became_peered":"2023-12-13T23:55:12.753618+0000","last_unstale":"2023-12-13T23:55:14.624938+0000","last_undegraded":"2023-12-13T23:55:14.624938+0000","last_fullsized":"2023-12-13T23:55:14.624938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:27:49.979804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623822+0000","last_change":"2023-12-13T23:55:12.755004+0000","last_active":"2023-12-13T23:55:14.623822+0000","last_peered":"2023-12-13T23:55:14.623822+0000","last_clean":"2023-12-13T23:55:14.623822+0000","last_became_active":"2023-12-13T23:55:12.754761+0000","last_became_peered":"2023-12-13T23:55:12.754761+0000","last_unstale":"2023-12-13T23:55:14.623822+0000","last_undegraded":"2023-12-13T23:55:14.623822+0000","last_fullsized":"2023-12-13T23:55:14.623822+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624213+0000","last_change":"2023-12-13T23:55:12.758828+0000","last_active":"2023-12-13T23:55:14.624213+0000","last_peered":"2023-12-13T23:55:14.624213+0000","last_clean":"2023-12-13T23:55:14.624213+0000","last_became_active":"2023-12-13T23:55:12.758520+0000","last_became_peered":"2023-12-13T23:55:12.758520+0000","last_unstale":"2023-12-13T23:55:14.624213+0000","last_undegraded":"2023-12-13T23:55:14.624213+0000","last_fullsized":"2023-12-13T23:55:14.624213+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740370+0000","last_change":"2023-12-13T23:55:12.745882+0000","last_active":"2023-12-13T23:55:13.740370+0000","last_peered":"2023-12-13T23:55:13.740370+0000","last_clean":"2023-12-13T23:55:13.740370+0000","last_became_active":"2023-12-13T23:55:12.744988+0000","last_became_peered":"2023-12-13T23:55:12.744988+0000","last_unstale":"2023-12-13T23:55:13.740370+0000","last_undegraded":"2023-12-13T23:55:13.740370+0000","last_fullsized":"2023-12-13T23:55:13.740370+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:57.837875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737851+0000","last_change":"2023-12-13T23:55:11.769713+0000","last_active":"2023-12-13T23:55:13.737851+0000","last_peered":"2023-12-13T23:55:13.737851+0000","last_clean":"2023-12-13T23:55:13.737851+0000","last_became_active":"2023-12-13T23:55:11.769540+0000","last_became_peered":"2023-12-13T23:55:11.769540+0000","last_unstale":"2023-12-13T23:55:13.737851+0000","last_undegraded":"2023-12-13T23:55:13.737851+0000","last_fullsized":"2023-12-13T23:55:13.737851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624177+0000","last_change":"2023-12-13T23:55:12.752916+0000","last_active":"2023-12-13T23:55:14.624177+0000","last_peered":"2023-12-13T23:55:14.624177+0000","last_clean":"2023-12-13T23:55:14.624177+0000","last_became_active":"2023-12-13T23:55:12.751877+0000","last_became_peered":"2023-12-13T23:55:12.751877+0000","last_unstale":"2023-12-13T23:55:14.624177+0000","last_undegraded":"2023-12-13T23:55:14.624177+0000","last_fullsized":"2023-12-13T23:55:14.624177+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624064+0000","last_change":"2023-12-13T23:55:11.767168+0000","last_active":"2023-12-13T23:55:14.624064+0000","last_peered":"2023-12-13T23:55:14.624064+0000","last_clean":"2023-12-13T23:55:14.624064+0000","last_became_active":"2023-12-13T23:55:11.766944+0000","last_became_peered":"2023-12-13T23:55:11.766944+0000","last_unstale":"2023-12-13T23:55:14.624064+0000","last_undegraded":"2023-12-13T23:55:14.624064+0000","last_fullsized":"2023-12-13T23:55:14.624064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737587+0000","last_change":"2023-12-13T23:55:11.764302+0000","last_active":"2023-12-13T23:55:13.737587+0000","last_peered":"2023-12-13T23:55:13.737587+0000","last_clean":"2023-12-13T23:55:13.737587+0000","last_became_active":"2023-12-13T23:55:11.763930+0000","last_became_peered":"2023-12-13T23:55:11.763930+0000","last_unstale":"2023-12-13T23:55:13.737587+0000","last_undegraded":"2023-12-13T23:55:13.737587+0000","last_fullsized":"2023-12-13T23:55:13.737587+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737560+0000","last_change":"2023-12-13T23:55:12.760219+0000","last_active":"2023-12-13T23:55:13.737560+0000","last_peered":"2023-12-13T23:55:13.737560+0000","last_clean":"2023-12-13T23:55:13.737560+0000","last_became_active":"2023-12-13T23:55:12.759890+0000","last_became_peered":"2023-12-13T23:55:12.759890+0000","last_unstale":"2023-12-13T23:55:13.737560+0000","last_undegraded":"2023-12-13T23:55:13.737560+0000","last_fullsized":"2023-12-13T23:55:13.737560+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624404+0000","last_change":"2023-12-13T23:55:11.763128+0000","last_active":"2023-12-13T23:55:14.624404+0000","last_peered":"2023-12-13T23:55:14.624404+0000","last_clean":"2023-12-13T23:55:14.624404+0000","last_became_active":"2023-12-13T23:55:11.762848+0000","last_became_peered":"2023-12-13T23:55:11.762848+0000","last_unstale":"2023-12-13T23:55:14.624404+0000","last_undegraded":"2023-12-13T23:55:14.624404+0000","last_fullsized":"2023-12-13T23:55:14.624404+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613510+0000","last_change":"2023-12-13T23:55:12.747367+0000","last_active":"2023-12-13T23:55:14.613510+0000","last_peered":"2023-12-13T23:55:14.613510+0000","last_clean":"2023-12-13T23:55:14.613510+0000","last_became_active":"2023-12-13T23:55:12.747074+0000","last_became_peered":"2023-12-13T23:55:12.747074+0000","last_unstale":"2023-12-13T23:55:14.613510+0000","last_undegraded":"2023-12-13T23:55:14.613510+0000","last_fullsized":"2023-12-13T23:55:14.613510+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:20.413240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624348+0000","last_change":"2023-12-13T23:55:11.766140+0000","last_active":"2023-12-13T23:55:14.624348+0000","last_peered":"2023-12-13T23:55:14.624348+0000","last_clean":"2023-12-13T23:55:14.624348+0000","last_became_active":"2023-12-13T23:55:11.765848+0000","last_became_peered":"2023-12-13T23:55:11.765848+0000","last_unstale":"2023-12-13T23:55:14.624348+0000","last_undegraded":"2023-12-13T23:55:14.624348+0000","last_fullsized":"2023-12-13T23:55:14.624348+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624786+0000","last_change":"2023-12-13T23:55:12.750678+0000","last_active":"2023-12-13T23:55:14.624786+0000","last_peered":"2023-12-13T23:55:14.624786+0000","last_clean":"2023-12-13T23:55:14.624786+0000","last_became_active":"2023-12-13T23:55:12.750200+0000","last_became_peered":"2023-12-13T23:55:12.750200+0000","last_unstale":"2023-12-13T23:55:14.624786+0000","last_undegraded":"2023-12-13T23:55:14.624786+0000","last_fullsized":"2023-12-13T23:55:14.624786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624863+0000","last_change":"2023-12-13T23:55:11.763844+0000","last_active":"2023-12-13T23:55:14.624863+0000","last_peered":"2023-12-13T23:55:14.624863+0000","last_clean":"2023-12-13T23:55:14.624863+0000","last_became_active":"2023-12-13T23:55:11.763574+0000","last_became_peered":"2023-12-13T23:55:11.763574+0000","last_unstale":"2023-12-13T23:55:14.624863+0000","last_undegraded":"2023-12-13T23:55:14.624863+0000","last_fullsized":"2023-12-13T23:55:14.624863+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:41:34.696890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624340+0000","last_change":"2023-12-13T23:55:12.754675+0000","last_active":"2023-12-13T23:55:14.624340+0000","last_peered":"2023-12-13T23:55:14.624340+0000","last_clean":"2023-12-13T23:55:14.624340+0000","last_became_active":"2023-12-13T23:55:12.753740+0000","last_became_peered":"2023-12-13T23:55:12.753740+0000","last_unstale":"2023-12-13T23:55:14.624340+0000","last_undegraded":"2023-12-13T23:55:14.624340+0000","last_fullsized":"2023-12-13T23:55:14.624340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613654+0000","last_change":"2023-12-13T23:55:11.765823+0000","last_active":"2023-12-13T23:55:14.613654+0000","last_peered":"2023-12-13T23:55:14.613654+0000","last_clean":"2023-12-13T23:55:14.613654+0000","last_became_active":"2023-12-13T23:55:11.765515+0000","last_became_peered":"2023-12-13T23:55:11.765515+0000","last_unstale":"2023-12-13T23:55:14.613654+0000","last_undegraded":"2023-12-13T23:55:14.613654+0000","last_fullsized":"2023-12-13T23:55:14.613654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741390+0000","last_change":"2023-12-13T23:55:12.750387+0000","last_active":"2023-12-13T23:55:13.741390+0000","last_peered":"2023-12-13T23:55:13.741390+0000","last_clean":"2023-12-13T23:55:13.741390+0000","last_became_active":"2023-12-13T23:55:12.749101+0000","last_became_peered":"2023-12-13T23:55:12.749101+0000","last_unstale":"2023-12-13T23:55:13.741390+0000","last_undegraded":"2023-12-13T23:55:13.741390+0000","last_fullsized":"2023-12-13T23:55:13.741390+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:20.838021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"20'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624137+0000","last_change":"2023-12-13T23:55:11.763071+0000","last_active":"2023-12-13T23:55:14.624137+0000","last_peered":"2023-12-13T23:55:14.624137+0000","last_clean":"2023-12-13T23:55:14.624137+0000","last_became_active":"2023-12-13T23:55:11.762774+0000","last_became_peered":"2023-12-13T23:55:11.762774+0000","last_unstale":"2023-12-13T23:55:14.624137+0000","last_undegraded":"2023-12-13T23:55:14.624137+0000","last_fullsized":"2023-12-13T23:55:14.624137+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740649+0000","last_change":"2023-12-13T23:55:12.747891+0000","last_active":"2023-12-13T23:55:13.740649+0000","last_peered":"2023-12-13T23:55:13.740649+0000","last_clean":"2023-12-13T23:55:13.740649+0000","last_became_active":"2023-12-13T23:55:12.747629+0000","last_became_peered":"2023-12-13T23:55:12.747629+0000","last_unstale":"2023-12-13T23:55:13.740649+0000","last_undegraded":"2023-12-13T23:55:13.740649+0000","last_fullsized":"2023-12-13T23:55:13.740649+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:01:19.278519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624263+0000","last_change":"2023-12-13T23:55:11.768277+0000","last_active":"2023-12-13T23:55:14.624263+0000","last_peered":"2023-12-13T23:55:14.624263+0000","last_clean":"2023-12-13T23:55:14.624263+0000","last_became_active":"2023-12-13T23:55:11.768039+0000","last_became_peered":"2023-12-13T23:55:11.768039+0000","last_unstale":"2023-12-13T23:55:14.624263+0000","last_undegraded":"2023-12-13T23:55:14.624263+0000","last_fullsized":"2023-12-13T23:55:14.624263+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624351+0000","last_change":"2023-12-13T23:55:12.758134+0000","last_active":"2023-12-13T23:55:14.624351+0000","last_peered":"2023-12-13T23:55:14.624351+0000","last_clean":"2023-12-13T23:55:14.624351+0000","last_became_active":"2023-12-13T23:55:12.757679+0000","last_became_peered":"2023-12-13T23:55:12.757679+0000","last_unstale":"2023-12-13T23:55:14.624351+0000","last_undegraded":"2023-12-13T23:55:14.624351+0000","last_fullsized":"2023-12-13T23:55:14.624351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625036+0000","last_change":"2023-12-13T23:55:11.765748+0000","last_active":"2023-12-13T23:55:14.625036+0000","last_peered":"2023-12-13T23:55:14.625036+0000","last_clean":"2023-12-13T23:55:14.625036+0000","last_became_active":"2023-12-13T23:55:11.765320+0000","last_became_peered":"2023-12-13T23:55:11.765320+0000","last_unstale":"2023-12-13T23:55:14.625036+0000","last_undegraded":"2023-12-13T23:55:14.625036+0000","last_fullsized":"2023-12-13T23:55:14.625036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740530+0000","last_change":"2023-12-13T23:55:12.747127+0000","last_active":"2023-12-13T23:55:13.740530+0000","last_peered":"2023-12-13T23:55:13.740530+0000","last_clean":"2023-12-13T23:55:13.740530+0000","last_became_active":"2023-12-13T23:55:12.746838+0000","last_became_peered":"2023-12-13T23:55:12.746838+0000","last_unstale":"2023-12-13T23:55:13.740530+0000","last_undegraded":"2023-12-13T23:55:13.740530+0000","last_fullsized":"2023-12-13T23:55:13.740530+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623961+0000","last_change":"2023-12-13T23:55:11.765098+0000","last_active":"2023-12-13T23:55:14.623961+0000","last_peered":"2023-12-13T23:55:14.623961+0000","last_clean":"2023-12-13T23:55:14.623961+0000","last_became_active":"2023-12-13T23:55:11.764746+0000","last_became_peered":"2023-12-13T23:55:11.764746+0000","last_unstale":"2023-12-13T23:55:14.623961+0000","last_undegraded":"2023-12-13T23:55:14.623961+0000","last_fullsized":"2023-12-13T23:55:14.623961+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:09:24.157031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624195+0000","last_change":"2023-12-13T23:55:12.759703+0000","last_active":"2023-12-13T23:55:14.624195+0000","last_peered":"2023-12-13T23:55:14.624195+0000","last_clean":"2023-12-13T23:55:14.624195+0000","last_became_active":"2023-12-13T23:55:12.759371+0000","last_became_peered":"2023-12-13T23:55:12.759371+0000","last_unstale":"2023-12-13T23:55:14.624195+0000","last_undegraded":"2023-12-13T23:55:14.624195+0000","last_fullsized":"2023-12-13T23:55:14.624195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:26.093689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623693+0000","last_change":"2023-12-13T23:55:11.757065+0000","last_active":"2023-12-13T23:55:14.623693+0000","last_peered":"2023-12-13T23:55:14.623693+0000","last_clean":"2023-12-13T23:55:14.623693+0000","last_became_active":"2023-12-13T23:55:11.756732+0000","last_became_peered":"2023-12-13T23:55:11.756732+0000","last_unstale":"2023-12-13T23:55:14.623693+0000","last_undegraded":"2023-12-13T23:55:14.623693+0000","last_fullsized":"2023-12-13T23:55:14.623693+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740938+0000","last_change":"2023-12-13T23:55:12.747168+0000","last_active":"2023-12-13T23:55:13.740938+0000","last_peered":"2023-12-13T23:55:13.740938+0000","last_clean":"2023-12-13T23:55:13.740938+0000","last_became_active":"2023-12-13T23:55:12.746860+0000","last_became_peered":"2023-12-13T23:55:12.746860+0000","last_unstale":"2023-12-13T23:55:13.740938+0000","last_undegraded":"2023-12-13T23:55:13.740938+0000","last_fullsized":"2023-12-13T23:55:13.740938+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:45:15.079845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740851+0000","last_change":"2023-12-13T23:55:11.765922+0000","last_active":"2023-12-13T23:55:13.740851+0000","last_peered":"2023-12-13T23:55:13.740851+0000","last_clean":"2023-12-13T23:55:13.740851+0000","last_became_active":"2023-12-13T23:55:11.765641+0000","last_became_peered":"2023-12-13T23:55:11.765641+0000","last_unstale":"2023-12-13T23:55:13.740851+0000","last_undegraded":"2023-12-13T23:55:13.740851+0000","last_fullsized":"2023-12-13T23:55:13.740851+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:07:21.569553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741605+0000","last_change":"2023-12-13T23:55:12.747433+0000","last_active":"2023-12-13T23:55:13.741605+0000","last_peered":"2023-12-13T23:55:13.741605+0000","last_clean":"2023-12-13T23:55:13.741605+0000","last_became_active":"2023-12-13T23:55:12.747169+0000","last_became_peered":"2023-12-13T23:55:12.747169+0000","last_unstale":"2023-12-13T23:55:13.741605+0000","last_undegraded":"2023-12-13T23:55:13.741605+0000","last_fullsized":"2023-12-13T23:55:13.741605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:03:57.776819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624072+0000","last_change":"2023-12-13T23:55:11.762985+0000","last_active":"2023-12-13T23:55:14.624072+0000","last_peered":"2023-12-13T23:55:14.624072+0000","last_clean":"2023-12-13T23:55:14.624072+0000","last_became_active":"2023-12-13T23:55:11.762654+0000","last_became_peered":"2023-12-13T23:55:11.762654+0000","last_unstale":"2023-12-13T23:55:14.624072+0000","last_undegraded":"2023-12-13T23:55:14.624072+0000","last_fullsized":"2023-12-13T23:55:14.624072+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:11.027745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741322+0000","last_change":"2023-12-13T23:55:12.756265+0000","last_active":"2023-12-13T23:55:13.741322+0000","last_peered":"2023-12-13T23:55:13.741322+0000","last_clean":"2023-12-13T23:55:13.741322+0000","last_became_active":"2023-12-13T23:55:12.756003+0000","last_became_peered":"2023-12-13T23:55:12.756003+0000","last_unstale":"2023-12-13T23:55:13.741322+0000","last_undegraded":"2023-12-13T23:55:13.741322+0000","last_fullsized":"2023-12-13T23:55:13.741322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:26:45.885932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613709+0000","last_change":"2023-12-13T23:55:11.766945+0000","last_active":"2023-12-13T23:55:14.613709+0000","last_peered":"2023-12-13T23:55:14.613709+0000","last_clean":"2023-12-13T23:55:14.613709+0000","last_became_active":"2023-12-13T23:55:11.766671+0000","last_became_peered":"2023-12-13T23:55:11.766671+0000","last_unstale":"2023-12-13T23:55:14.613709+0000","last_undegraded":"2023-12-13T23:55:14.613709+0000","last_fullsized":"2023-12-13T23:55:14.613709+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624646+0000","last_change":"2023-12-13T23:55:12.752034+0000","last_active":"2023-12-13T23:55:14.624646+0000","last_peered":"2023-12-13T23:55:14.624646+0000","last_clean":"2023-12-13T23:55:14.624646+0000","last_became_active":"2023-12-13T23:55:12.751764+0000","last_became_peered":"2023-12-13T23:55:12.751764+0000","last_unstale":"2023-12-13T23:55:14.624646+0000","last_undegraded":"2023-12-13T23:55:14.624646+0000","last_fullsized":"2023-12-13T23:55:14.624646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623912+0000","last_change":"2023-12-13T23:55:11.765879+0000","last_active":"2023-12-13T23:55:14.623912+0000","last_peered":"2023-12-13T23:55:14.623912+0000","last_clean":"2023-12-13T23:55:14.623912+0000","last_became_active":"2023-12-13T23:55:11.765607+0000","last_became_peered":"2023-12-13T23:55:11.765607+0000","last_unstale":"2023-12-13T23:55:14.623912+0000","last_undegraded":"2023-12-13T23:55:14.623912+0000","last_fullsized":"2023-12-13T23:55:14.623912+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:32:40.262294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740784+0000","last_change":"2023-12-13T23:55:12.756349+0000","last_active":"2023-12-13T23:55:13.740784+0000","last_peered":"2023-12-13T23:55:13.740784+0000","last_clean":"2023-12-13T23:55:13.740784+0000","last_became_active":"2023-12-13T23:55:12.756033+0000","last_became_peered":"2023-12-13T23:55:12.756033+0000","last_unstale":"2023-12-13T23:55:13.740784+0000","last_undegraded":"2023-12-13T23:55:13.740784+0000","last_fullsized":"2023-12-13T23:55:13.740784+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:12:11.614423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":165,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-12-13T23:55:12.739242+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:55:12.739242+0000","last_peered":"2023-12-13T23:55:12.739242+0000","last_clean":"2023-12-13T23:55:12.739242+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:55:12.739242+0000","last_undegraded":"2023-12-13T23:55:12.739242+0000","last_fullsized":"2023-12-13T23:55:12.739242+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624508+0000","last_change":"2023-12-13T23:55:11.758971+0000","last_active":"2023-12-13T23:55:14.624508+0000","last_peered":"2023-12-13T23:55:14.624508+0000","last_clean":"2023-12-13T23:55:14.624508+0000","last_became_active":"2023-12-13T23:55:11.758594+0000","last_became_peered":"2023-12-13T23:55:11.758594+0000","last_unstale":"2023-12-13T23:55:14.624508+0000","last_undegraded":"2023-12-13T23:55:14.624508+0000","last_fullsized":"2023-12-13T23:55:14.624508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:26:19.833569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737699+0000","last_change":"2023-12-13T23:55:12.755959+0000","last_active":"2023-12-13T23:55:13.737699+0000","last_peered":"2023-12-13T23:55:13.737699+0000","last_clean":"2023-12-13T23:55:13.737699+0000","last_became_active":"2023-12-13T23:55:12.755662+0000","last_became_peered":"2023-12-13T23:55:12.755662+0000","last_unstale":"2023-12-13T23:55:13.737699+0000","last_undegraded":"2023-12-13T23:55:13.737699+0000","last_fullsized":"2023-12-13T23:55:13.737699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740316+0000","last_change":"2023-12-13T23:55:11.767937+0000","last_active":"2023-12-13T23:55:13.740316+0000","last_peered":"2023-12-13T23:55:13.740316+0000","last_clean":"2023-12-13T23:55:13.740316+0000","last_became_active":"2023-12-13T23:55:11.767613+0000","last_became_peered":"2023-12-13T23:55:11.767613+0000","last_unstale":"2023-12-13T23:55:13.740316+0000","last_undegraded":"2023-12-13T23:55:13.740316+0000","last_fullsized":"2023-12-13T23:55:13.740316+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:53.188642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624112+0000","last_change":"2023-12-13T23:55:12.757755+0000","last_active":"2023-12-13T23:55:14.624112+0000","last_peered":"2023-12-13T23:55:14.624112+0000","last_clean":"2023-12-13T23:55:14.624112+0000","last_became_active":"2023-12-13T23:55:12.757456+0000","last_became_peered":"2023-12-13T23:55:12.757456+0000","last_unstale":"2023-12-13T23:55:14.624112+0000","last_undegraded":"2023-12-13T23:55:14.624112+0000","last_fullsized":"2023-12-13T23:55:14.624112+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:46.296822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.623940+0000","last_change":"2023-12-13T23:55:11.755829+0000","last_active":"2023-12-13T23:55:14.623940+0000","last_peered":"2023-12-13T23:55:14.623940+0000","last_clean":"2023-12-13T23:55:14.623940+0000","last_became_active":"2023-12-13T23:55:11.755573+0000","last_became_peered":"2023-12-13T23:55:11.755573+0000","last_unstale":"2023-12-13T23:55:14.623940+0000","last_undegraded":"2023-12-13T23:55:14.623940+0000","last_fullsized":"2023-12-13T23:55:14.623940+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:59:56.833424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613504+0000","last_change":"2023-12-13T23:55:12.749732+0000","last_active":"2023-12-13T23:55:14.613504+0000","last_peered":"2023-12-13T23:55:14.613504+0000","last_clean":"2023-12-13T23:55:14.613504+0000","last_became_active":"2023-12-13T23:55:12.749473+0000","last_became_peered":"2023-12-13T23:55:12.749473+0000","last_unstale":"2023-12-13T23:55:14.613504+0000","last_undegraded":"2023-12-13T23:55:14.613504+0000","last_fullsized":"2023-12-13T23:55:14.613504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625041+0000","last_change":"2023-12-13T23:55:11.765970+0000","last_active":"2023-12-13T23:55:14.625041+0000","last_peered":"2023-12-13T23:55:14.625041+0000","last_clean":"2023-12-13T23:55:14.625041+0000","last_became_active":"2023-12-13T23:55:11.765629+0000","last_became_peered":"2023-12-13T23:55:11.765629+0000","last_unstale":"2023-12-13T23:55:14.625041+0000","last_undegraded":"2023-12-13T23:55:14.625041+0000","last_fullsized":"2023-12-13T23:55:14.625041+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741379+0000","last_change":"2023-12-13T23:55:12.758873+0000","last_active":"2023-12-13T23:55:13.741379+0000","last_peered":"2023-12-13T23:55:13.741379+0000","last_clean":"2023-12-13T23:55:13.741379+0000","last_became_active":"2023-12-13T23:55:12.758618+0000","last_became_peered":"2023-12-13T23:55:12.758618+0000","last_unstale":"2023-12-13T23:55:13.741379+0000","last_undegraded":"2023-12-13T23:55:13.741379+0000","last_fullsized":"2023-12-13T23:55:13.741379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:45.977596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613597+0000","last_change":"2023-12-13T23:55:11.762944+0000","last_active":"2023-12-13T23:55:14.613597+0000","last_peered":"2023-12-13T23:55:14.613597+0000","last_clean":"2023-12-13T23:55:14.613597+0000","last_became_active":"2023-12-13T23:55:11.762691+0000","last_became_peered":"2023-12-13T23:55:11.762691+0000","last_unstale":"2023-12-13T23:55:14.613597+0000","last_undegraded":"2023-12-13T23:55:14.613597+0000","last_fullsized":"2023-12-13T23:55:14.613597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.625017+0000","last_change":"2023-12-13T23:55:12.755585+0000","last_active":"2023-12-13T23:55:14.625017+0000","last_peered":"2023-12-13T23:55:14.625017+0000","last_clean":"2023-12-13T23:55:14.625017+0000","last_became_active":"2023-12-13T23:55:12.755284+0000","last_became_peered":"2023-12-13T23:55:12.755284+0000","last_unstale":"2023-12-13T23:55:14.625017+0000","last_undegraded":"2023-12-13T23:55:14.625017+0000","last_fullsized":"2023-12-13T23:55:14.625017+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:44:41.447192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741015+0000","last_change":"2023-12-13T23:55:11.770291+0000","last_active":"2023-12-13T23:55:13.741015+0000","last_peered":"2023-12-13T23:55:13.741015+0000","last_clean":"2023-12-13T23:55:13.741015+0000","last_became_active":"2023-12-13T23:55:11.769672+0000","last_became_peered":"2023-12-13T23:55:11.769672+0000","last_unstale":"2023-12-13T23:55:13.741015+0000","last_undegraded":"2023-12-13T23:55:13.741015+0000","last_fullsized":"2023-12-13T23:55:13.741015+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:40:19.408788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613615+0000","last_change":"2023-12-13T23:55:12.749467+0000","last_active":"2023-12-13T23:55:14.613615+0000","last_peered":"2023-12-13T23:55:14.613615+0000","last_clean":"2023-12-13T23:55:14.613615+0000","last_became_active":"2023-12-13T23:55:12.749185+0000","last_became_peered":"2023-12-13T23:55:12.749185+0000","last_unstale":"2023-12-13T23:55:14.613615+0000","last_undegraded":"2023-12-13T23:55:14.613615+0000","last_fullsized":"2023-12-13T23:55:14.613615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:15.121343+0000","last_change":"2023-12-13T23:55:11.763643+0000","last_active":"2023-12-13T23:55:15.121343+0000","last_peered":"2023-12-13T23:55:15.121343+0000","last_clean":"2023-12-13T23:55:15.121343+0000","last_became_active":"2023-12-13T23:55:11.763378+0000","last_became_peered":"2023-12-13T23:55:11.763378+0000","last_unstale":"2023-12-13T23:55:15.121343+0000","last_undegraded":"2023-12-13T23:55:15.121343+0000","last_fullsized":"2023-12-13T23:55:15.121343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737984+0000","last_change":"2023-12-13T23:55:12.752448+0000","last_active":"2023-12-13T23:55:13.737984+0000","last_peered":"2023-12-13T23:55:13.737984+0000","last_clean":"2023-12-13T23:55:13.737984+0000","last_became_active":"2023-12-13T23:55:12.752155+0000","last_became_peered":"2023-12-13T23:55:12.752155+0000","last_unstale":"2023-12-13T23:55:13.737984+0000","last_undegraded":"2023-12-13T23:55:13.737984+0000","last_fullsized":"2023-12-13T23:55:13.737984+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:03:20.568951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624790+0000","last_change":"2023-12-13T23:55:11.768156+0000","last_active":"2023-12-13T23:55:14.624790+0000","last_peered":"2023-12-13T23:55:14.624790+0000","last_clean":"2023-12-13T23:55:14.624790+0000","last_became_active":"2023-12-13T23:55:11.767878+0000","last_became_peered":"2023-12-13T23:55:11.767878+0000","last_unstale":"2023-12-13T23:55:14.624790+0000","last_undegraded":"2023-12-13T23:55:14.624790+0000","last_fullsized":"2023-12-13T23:55:14.624790+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:56:23.382918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624226+0000","last_change":"2023-12-13T23:55:12.751906+0000","last_active":"2023-12-13T23:55:14.624226+0000","last_peered":"2023-12-13T23:55:14.624226+0000","last_clean":"2023-12-13T23:55:14.624226+0000","last_became_active":"2023-12-13T23:55:12.751468+0000","last_became_peered":"2023-12-13T23:55:12.751468+0000","last_unstale":"2023-12-13T23:55:14.624226+0000","last_undegraded":"2023-12-13T23:55:14.624226+0000","last_fullsized":"2023-12-13T23:55:14.624226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:44.400998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740225+0000","last_change":"2023-12-13T23:55:11.768417+0000","last_active":"2023-12-13T23:55:13.740225+0000","last_peered":"2023-12-13T23:55:13.740225+0000","last_clean":"2023-12-13T23:55:13.740225+0000","last_became_active":"2023-12-13T23:55:11.768141+0000","last_became_peered":"2023-12-13T23:55:11.768141+0000","last_unstale":"2023-12-13T23:55:13.740225+0000","last_undegraded":"2023-12-13T23:55:13.740225+0000","last_fullsized":"2023-12-13T23:55:13.740225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:48:23.286885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740516+0000","last_change":"2023-12-13T23:55:12.751650+0000","last_active":"2023-12-13T23:55:13.740516+0000","last_peered":"2023-12-13T23:55:13.740516+0000","last_clean":"2023-12-13T23:55:13.740516+0000","last_became_active":"2023-12-13T23:55:12.749884+0000","last_became_peered":"2023-12-13T23:55:12.749884+0000","last_unstale":"2023-12-13T23:55:13.740516+0000","last_undegraded":"2023-12-13T23:55:13.740516+0000","last_fullsized":"2023-12-13T23:55:13.740516+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:19:12.032197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613607+0000","last_change":"2023-12-13T23:55:11.761362+0000","last_active":"2023-12-13T23:55:14.613607+0000","last_peered":"2023-12-13T23:55:14.613607+0000","last_clean":"2023-12-13T23:55:14.613607+0000","last_became_active":"2023-12-13T23:55:11.761091+0000","last_became_peered":"2023-12-13T23:55:11.761091+0000","last_unstale":"2023-12-13T23:55:14.613607+0000","last_undegraded":"2023-12-13T23:55:14.613607+0000","last_fullsized":"2023-12-13T23:55:14.613607+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:38:45.292370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737621+0000","last_change":"2023-12-13T23:55:12.747019+0000","last_active":"2023-12-13T23:55:13.737621+0000","last_peered":"2023-12-13T23:55:13.737621+0000","last_clean":"2023-12-13T23:55:13.737621+0000","last_became_active":"2023-12-13T23:55:12.746720+0000","last_became_peered":"2023-12-13T23:55:12.746720+0000","last_unstale":"2023-12-13T23:55:13.737621+0000","last_undegraded":"2023-12-13T23:55:13.737621+0000","last_fullsized":"2023-12-13T23:55:13.737621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:50:29.832777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.613672+0000","last_change":"2023-12-13T23:55:11.761235+0000","last_active":"2023-12-13T23:55:14.613672+0000","last_peered":"2023-12-13T23:55:14.613672+0000","last_clean":"2023-12-13T23:55:14.613672+0000","last_became_active":"2023-12-13T23:55:11.760961+0000","last_became_peered":"2023-12-13T23:55:11.760961+0000","last_unstale":"2023-12-13T23:55:14.613672+0000","last_undegraded":"2023-12-13T23:55:14.613672+0000","last_fullsized":"2023-12-13T23:55:14.613672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:58:24.646906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737873+0000","last_change":"2023-12-13T23:55:12.757945+0000","last_active":"2023-12-13T23:55:13.737873+0000","last_peered":"2023-12-13T23:55:13.737873+0000","last_clean":"2023-12-13T23:55:13.737873+0000","last_became_active":"2023-12-13T23:55:12.757685+0000","last_became_peered":"2023-12-13T23:55:12.757685+0000","last_unstale":"2023-12-13T23:55:13.737873+0000","last_undegraded":"2023-12-13T23:55:13.737873+0000","last_fullsized":"2023-12-13T23:55:13.737873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:42:25.865126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740608+0000","last_change":"2023-12-13T23:55:11.769857+0000","last_active":"2023-12-13T23:55:13.740608+0000","last_peered":"2023-12-13T23:55:13.740608+0000","last_clean":"2023-12-13T23:55:13.740608+0000","last_became_active":"2023-12-13T23:55:11.769503+0000","last_became_peered":"2023-12-13T23:55:11.769503+0000","last_unstale":"2023-12-13T23:55:13.740608+0000","last_undegraded":"2023-12-13T23:55:13.740608+0000","last_fullsized":"2023-12-13T23:55:13.740608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:25.089237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624730+0000","last_change":"2023-12-13T23:55:12.747991+0000","last_active":"2023-12-13T23:55:14.624730+0000","last_peered":"2023-12-13T23:55:14.624730+0000","last_clean":"2023-12-13T23:55:14.624730+0000","last_became_active":"2023-12-13T23:55:12.746787+0000","last_became_peered":"2023-12-13T23:55:12.746787+0000","last_unstale":"2023-12-13T23:55:14.624730+0000","last_undegraded":"2023-12-13T23:55:14.624730+0000","last_fullsized":"2023-12-13T23:55:14.624730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624966+0000","last_change":"2023-12-13T23:55:12.755883+0000","last_active":"2023-12-13T23:55:14.624966+0000","last_peered":"2023-12-13T23:55:14.624966+0000","last_clean":"2023-12-13T23:55:14.624966+0000","last_became_active":"2023-12-13T23:55:12.755573+0000","last_became_peered":"2023-12-13T23:55:12.755573+0000","last_unstale":"2023-12-13T23:55:14.624966+0000","last_undegraded":"2023-12-13T23:55:14.624966+0000","last_fullsized":"2023-12-13T23:55:14.624966+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:36:53.394180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.741033+0000","last_change":"2023-12-13T23:55:11.769532+0000","last_active":"2023-12-13T23:55:13.741033+0000","last_peered":"2023-12-13T23:55:13.741033+0000","last_clean":"2023-12-13T23:55:13.741033+0000","last_became_active":"2023-12-13T23:55:11.769251+0000","last_became_peered":"2023-12-13T23:55:11.769251+0000","last_unstale":"2023-12-13T23:55:13.741033+0000","last_undegraded":"2023-12-13T23:55:13.741033+0000","last_fullsized":"2023-12-13T23:55:13.741033+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:35:44.973144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624446+0000","last_change":"2023-12-13T23:55:12.757891+0000","last_active":"2023-12-13T23:55:14.624446+0000","last_peered":"2023-12-13T23:55:14.624446+0000","last_clean":"2023-12-13T23:55:14.624446+0000","last_became_active":"2023-12-13T23:55:12.757643+0000","last_became_peered":"2023-12-13T23:55:12.757643+0000","last_unstale":"2023-12-13T23:55:14.624446+0000","last_undegraded":"2023-12-13T23:55:14.624446+0000","last_fullsized":"2023-12-13T23:55:14.624446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:46:49.006604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.740911+0000","last_change":"2023-12-13T23:55:11.769811+0000","last_active":"2023-12-13T23:55:13.740911+0000","last_peered":"2023-12-13T23:55:13.740911+0000","last_clean":"2023-12-13T23:55:13.740911+0000","last_became_active":"2023-12-13T23:55:11.769483+0000","last_became_peered":"2023-12-13T23:55:11.769483+0000","last_unstale":"2023-12-13T23:55:13.740911+0000","last_undegraded":"2023-12-13T23:55:13.740911+0000","last_fullsized":"2023-12-13T23:55:13.740911+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:58:43.396546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624275+0000","last_change":"2023-12-13T23:55:12.758511+0000","last_active":"2023-12-13T23:55:14.624275+0000","last_peered":"2023-12-13T23:55:14.624275+0000","last_clean":"2023-12-13T23:55:14.624275+0000","last_became_active":"2023-12-13T23:55:12.757824+0000","last_became_peered":"2023-12-13T23:55:12.757824+0000","last_unstale":"2023-12-13T23:55:14.624275+0000","last_undegraded":"2023-12-13T23:55:14.624275+0000","last_fullsized":"2023-12-13T23:55:14.624275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:32:55.745178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737890+0000","last_change":"2023-12-13T23:55:11.761808+0000","last_active":"2023-12-13T23:55:13.737890+0000","last_peered":"2023-12-13T23:55:13.737890+0000","last_clean":"2023-12-13T23:55:13.737890+0000","last_became_active":"2023-12-13T23:55:11.761559+0000","last_became_peered":"2023-12-13T23:55:11.761559+0000","last_unstale":"2023-12-13T23:55:13.737890+0000","last_undegraded":"2023-12-13T23:55:13.737890+0000","last_fullsized":"2023-12-13T23:55:13.737890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:13.737814+0000","last_change":"2023-12-13T23:55:12.748462+0000","last_active":"2023-12-13T23:55:13.737814+0000","last_peered":"2023-12-13T23:55:13.737814+0000","last_clean":"2023-12-13T23:55:13.737814+0000","last_became_active":"2023-12-13T23:55:12.748166+0000","last_became_peered":"2023-12-13T23:55:12.748166+0000","last_unstale":"2023-12-13T23:55:13.737814+0000","last_undegraded":"2023-12-13T23:55:13.737814+0000","last_fullsized":"2023-12-13T23:55:13.737814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:11.740391+0000","last_clean_scrub_stamp":"2023-12-13T23:55:11.740391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:54.193094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"20'4","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-12-13T23:55:14.624331+0000","last_change":"2023-12-13T23:55:11.771111+0000","last_active":"2023-12-13T23:55:14.624331+0000","last_peered":"2023-12-13T23:55:14.624331+0000","last_clean":"2023-12-13T23:55:14.624331+0000","last_became_active":"2023-12-13T23:55:11.770795+0000","last_became_peered":"2023-12-13T23:55:11.770795+0000","last_unstale":"2023-12-13T23:55:14.624331+0000","last_undegraded":"2023-12-13T23:55:14.624331+0000","last_fullsized":"2023-12-13T23:55:14.624331+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:10.735939+0000","last_clean_scrub_stamp":"2023-12-13T23:55:10.735939+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:33:47.748881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":815104,"data_stored":666531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":75671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":75707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56499999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":229376,"data_stored":77193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29199999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":75581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.248}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":225280,"data_stored":75741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":666587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0449999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:55:24.675 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-13T23:55:24.675 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-13T23:55:24.675 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-13T23:55:24.676 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-13T23:55:24.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.814+0000 7f1cd6f0f640 1 Processor -- start 2023-12-13T23:55:24.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.814+0000 7f1cd6f0f640 1 -- start start 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd6f0f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 0x7f1cd015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd6f0f640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cd015e730 0x7f1cd015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd6f0f640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd015e2a0 con 0x7f1cd0143490 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd6f0f640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd015b520 con 0x7f1cd015e730 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd6f0f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd0143870 con 0x7f1cd00b1400 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd54ac640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cd015e730 0x7f1cd015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd4cab640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f1cd0143490 0x7f1cd00b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43690/0 (socket says 172.21.15.124:43690) 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd4cab640 1 -- 172.21.15.124:0/3401606252 learned_addr learned my addr 172.21.15.124:0/3401606252 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cc7fff640 1 --2- 172.21.15.124:0/3401606252 >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 0x7f1cd015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd54ac640 1 -- 172.21.15.124:0/3401606252 >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 msgr2=0x7f1cd015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd54ac640 1 --2- 172.21.15.124:0/3401606252 >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 0x7f1cd015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd54ac640 1 -- 172.21.15.124:0/3401606252 >> v1:172.21.15.124:6789/0 conn(0x7f1cd0143490 legacy=0x7f1cd00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cd54ac640 1 -- 172.21.15.124:0/3401606252 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd015b240 con 0x7f1cd015e730 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.815+0000 7f1cc7fff640 1 --2- 172.21.15.124:0/3401606252 >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 0x7f1cd015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd54ac640 1 --2- 172.21.15.124:0/3401606252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd015aa20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00aeca0 tx=0x7f1cc802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9219892ccc960f6a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cc77fe640 1 -- 172.21.15.124:0/3401606252 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc802cc80 con 0x7f1cd015e730 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cc77fe640 1 -- 172.21.15.124:0/3401606252 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc802cde0 con 0x7f1cd015e730 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cc77fe640 1 -- 172.21.15.124:0/3401606252 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8034660 con 0x7f1cd015e730 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/3401606252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 msgr2=0x7f1cd015aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/3401606252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd015aa20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1cd00aeca0 tx=0x7f1cc802bed0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/3401606252 shutdown_connections 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/3401606252 >> v2:172.21.15.203:3300/0 conn(0x7f1cd00b1400 0x7f1cd015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/3401606252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd015aa20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/3401606252 >> 172.21.15.124:0/3401606252 conn(0x7f1cd006a4d0 msgr2=0x7f1cd006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:24.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/3401606252 shutdown_connections 2023-12-13T23:55:24.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.816+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/3401606252 wait complete. 2023-12-13T23:55:24.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.817+0000 7f1cd6f0f640 1 Processor -- start 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 -- start start 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 0x7f1cd013e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 0x7f1cd013d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd013fc00 con 0x7f1cd015e730 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd01e57e0 con 0x7f1cd0143490 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd6f0f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd013fe30 con 0x7f1cd00b1400 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd54ac640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd54ac640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57502/0 (socket says 172.21.15.124:57502) 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.818+0000 7f1cd54ac640 1 -- 172.21.15.124:0/1575956946 learned_addr learned my addr 172.21.15.124:0/1575956946 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cc7fff640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 0x7f1cd013d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd4cab640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 0x7f1cd013e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 -- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 msgr2=0x7f1cd013e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 0x7f1cd013e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 -- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 msgr2=0x7f1cd013d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 0x7f1cd013d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 -- 172.21.15.124:0/1575956946 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc8009a70 con 0x7f1cd015e730 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cc7fff640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 0x7f1cd013d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd4cab640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 0x7f1cd013e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd54ac640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009e80 tx=0x7f1cc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8040650 con 0x7f1cd015e730 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc80407b0 con 0x7f1cd015e730 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc8040b60 con 0x7f1cd015e730 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd0140110 con 0x7f1cd015e730 2023-12-13T23:55:24.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.819+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd00a92e0 con 0x7f1cd015e730 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.820+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd0069400 con 0x7f1cd015e730 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.820+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1cc8040cc0 con 0x7f1cd015e730 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.821+0000 7f1cc57fa640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 0x7f1cbc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.821+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1cc80ac8a0 con 0x7f1cd015e730 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.821+0000 7f1cd4cab640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 0x7f1cbc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:24.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.821+0000 7f1cd4cab640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 0x7f1cbc06d680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005880 tx=0x7f1ccc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:24.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:24.825+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1cc8077690 con 0x7f1cd015e730 2023-12-13T23:55:25.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.028+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1cd00ae310 con 0x7f1cd015e730 2023-12-13T23:55:25.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.028+0000 7f1cc57fa640 1 -- 172.21.15.124:0/1575956946 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1cc807acb0 con 0x7f1cd015e730 2023-12-13T23:55:25.026 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:25.026 INFO:teuthology.orchestra.run.smithi124.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-13T23:55:25.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 msgr2=0x7f1cbc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 0x7f1cbc06d680 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc005880 tx=0x7f1ccc009040 comp rx=0 tx=0).stop 2023-12-13T23:55:25.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 msgr2=0x7f1cd00a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1cc8009e80 tx=0x7f1cc8004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:25.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 shutdown_connections 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cbc06b1c0 0x7f1cbc06d680 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cd0143490 0x7f1cd013d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cd015e730 0x7f1cd00a84d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 --2- 172.21.15.124:0/1575956946 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cd00b1400 0x7f1cd013e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 >> 172.21.15.124:0/1575956946 conn(0x7f1cd006a4d0 msgr2=0x7f1cd0141690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 shutdown_connections 2023-12-13T23:55:25.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.030+0000 7f1cd6f0f640 1 -- 172.21.15.124:0/1575956946 wait complete. 2023-12-13T23:55:25.039 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-13T23:55:25.039 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:25.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.141+0000 7f89b15d7640 1 Processor -- start 2023-12-13T23:55:25.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.141+0000 7f89b15d7640 1 -- start start 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89b15d7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f89ac0d2910 0x7f89ac0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89b15d7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f89ac0d3730 0x7f89ac0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89b15d7640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89ac0b2b10 con 0x7f89ac15d330 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89b15d7640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0d3bc0 con 0x7f89ac0d3730 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89b15d7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89ac0d1590 con 0x7f89ac0d2910 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f89ac0d3730 0x7f89ac0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f89ac0d3730 0x7f89ac0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57512/0 (socket says 172.21.15.124:57512) 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2590529622 learned_addr learned my addr 172.21.15.124:0/2590529622 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89aaffd640 1 --1- 172.21.15.124:0/2590529622 >> v1:172.21.15.124:6789/0 conn(0x7f89ac15d330 0x7f89ac0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43700/0 (socket says 172.21.15.124:43700) 2023-12-13T23:55:25.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2590529622 >> v2:172.21.15.203:3300/0 conn(0x7f89ac0d2910 msgr2=0x7f89ac0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89a27fc640 1 --2- 172.21.15.124:0/2590529622 >> v2:172.21.15.203:3300/0 conn(0x7f89ac0d2910 0x7f89ac0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 --2- 172.21.15.124:0/2590529622 >> v2:172.21.15.203:3300/0 conn(0x7f89ac0d2910 0x7f89ac0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89aa7fc640 1 -- 172.21.15.124:0/2590529622 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691202374 0 0) 0x7f89ac0b2b10 con 0x7f89ac15d330 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2590529622 >> v1:172.21.15.124:6789/0 conn(0x7f89ac15d330 legacy=0x7f89ac0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.142+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2590529622 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac0d1870 con 0x7f89ac0d3730 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89ab7fe640 1 --2- 172.21.15.124:0/2590529622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac0d3730 0x7f89ac0b6390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f89ac0d1540 tx=0x7f89a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73c244825f385435 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89aa7fc640 1 -- 172.21.15.124:0/2590529622 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a402ca80 con 0x7f89ac0d3730 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89aa7fc640 1 -- 172.21.15.124:0/2590529622 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a402cbe0 con 0x7f89ac0d3730 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89aa7fc640 1 -- 172.21.15.124:0/2590529622 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a40343f0 con 0x7f89ac0d3730 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 -- 172.21.15.124:0/2590529622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac0d3730 msgr2=0x7f89ac0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2590529622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac0d3730 0x7f89ac0b6390 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f89ac0d1540 tx=0x7f89a402bee0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 -- 172.21.15.124:0/2590529622 shutdown_connections 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2590529622 >> v2:172.21.15.203:3300/0 conn(0x7f89ac0d2910 0x7f89ac0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2590529622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac0d3730 0x7f89ac0b6390 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 -- 172.21.15.124:0/2590529622 >> 172.21.15.124:0/2590529622 conn(0x7f89ac072020 msgr2=0x7f89ac072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.143+0000 7f89b15d7640 1 -- 172.21.15.124:0/2590529622 shutdown_connections 2023-12-13T23:55:25.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.144+0000 7f89b15d7640 1 -- 172.21.15.124:0/2590529622 wait complete. 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 Processor -- start 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 -- start start 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ac0d2910 0x7f89ac1530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89ac0d3730 0x7f89ac15f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac157940 con 0x7f89ac15d330 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1607b0 con 0x7f89ac0d2910 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.145+0000 7f89b15d7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1601f0 con 0x7f89ac0d3730 2023-12-13T23:55:25.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57524/0 (socket says 172.21.15.124:57524) 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2953711491 learned_addr learned my addr 172.21.15.124:0/2953711491 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89aaffd640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ac0d2910 0x7f89ac1530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89a27fc640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89ac0d3730 0x7f89ac15f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89ac0d3730 msgr2=0x7f89ac15f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89ac0d3730 0x7f89ac15f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ac0d2910 msgr2=0x7f89ac1530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ac0d2910 0x7f89ac1530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 -- 172.21.15.124:0/2953711491 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4009b30 con 0x7f89ac15d330 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89ab7fe640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009f40 tx=0x7f89a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a4009510 con 0x7f89ac15d330 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89a403e450 con 0x7f89ac15d330 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac1604d0 con 0x7f89ac15d330 2023-12-13T23:55:25.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a403e800 con 0x7f89ac15d330 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.146+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac15ff00 con 0x7f89ac15d330 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.147+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac157a80 con 0x7f89ac15d330 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.148+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f89a402cd50 con 0x7f89ac15d330 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.148+0000 7f89a3fff640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 0x7f898c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.148+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89a40ac3f0 con 0x7f89ac15d330 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.148+0000 7f89aaffd640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 0x7f898c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.149+0000 7f89aaffd640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 0x7f898c06d680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f899c000f60 tx=0x7f899c007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.152+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f89a4077260 con 0x7f89ac15d330 2023-12-13T23:55:25.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.321+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89ac069400 con 0x7f89ac15d330 2023-12-13T23:55:25.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.322+0000 7f89a3fff640 1 -- 172.21.15.124:0/2953711491 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f89a407a880 con 0x7f89ac15d330 2023-12-13T23:55:25.321 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:25.321 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 msgr2=0x7f898c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 0x7f898c06d680 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f899c000f60 tx=0x7f899c007d50 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 msgr2=0x7f89ac15b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89a4009f40 tx=0x7f89a4004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 shutdown_connections 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f898c06b1c0 0x7f898c06d680 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89ac15d330 0x7f89ac15b3e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89ac0d3730 0x7f89ac15f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 --2- 172.21.15.124:0/2953711491 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89ac0d2910 0x7f89ac1530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 >> 172.21.15.124:0/2953711491 conn(0x7f89ac072020 msgr2=0x7f89ac072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 shutdown_connections 2023-12-13T23:55:25.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.324+0000 7f89b15d7640 1 -- 172.21.15.124:0/2953711491 wait complete. 2023-12-13T23:55:25.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:25.334 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:25.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 Processor -- start 2023-12-13T23:55:25.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 -- start start 2023-12-13T23:55:25.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f97400cebb0 0x7f97400c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400c7ee0 con 0x7f97400c63c0 2023-12-13T23:55:25.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97400cf330 con 0x7f974015bbe0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.438+0000 7f9746961640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97400c7920 con 0x7f97400cebb0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f9744efe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f97400c63c0 0x7f974015f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43716/0 (socket says 172.21.15.124:43716) 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f9744efe640 1 -- 172.21.15.124:0/1968430469 learned_addr learned my addr 172.21.15.124:0/1968430469 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300616744 0 0) 0x7f97400c7ee0 con 0x7f97400c63c0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973f7fe640 1 --2- 172.21.15.124:0/1968430469 >> v2:172.21.15.203:3300/0 conn(0x7f97400cebb0 0x7f97400c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9728003520 con 0x7f97400c63c0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495424070 0 0) 0x7f9728003520 con 0x7f97400c63c0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 >> v2:172.21.15.203:3300/0 conn(0x7f97400cebb0 msgr2=0x7f97400c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 --2- 172.21.15.124:0/1968430469 >> v2:172.21.15.203:3300/0 conn(0x7f97400cebb0 0x7f97400c5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 >> v1:172.21.15.203:6789/0 conn(0x7f974015bbe0 legacy=0x7f974015bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400c7c00 con 0x7f97400c63c0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f97380019f0 con 0x7f97400c63c0 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 >> v1:172.21.15.124:6789/0 conn(0x7f97400c63c0 legacy=0x7f974015f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9728003970 0x7f9728003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97280047c0 0x7f9728004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 0x7f972800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97400c7920 con 0x7f97280052a0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972800c170 con 0x7f97280047c0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972800c330 con 0x7f9728003970 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f9744efe640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 0x7f972800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973f7fe640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97280047c0 0x7f9728004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.439+0000 7f973ffff640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9728003970 0x7f9728003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 -- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9728003970 msgr2=0x7f9728003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9728003970 0x7f9728003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 -- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97280047c0 msgr2=0x7f9728004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97280047c0 0x7f9728004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 -- 172.21.15.124:0/1968430469 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9728003520 con 0x7f97280052a0 2023-12-13T23:55:25.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 -- 172.21.15.124:0/1968430469 reap_dead start 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f9744efe640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 0x7f972800b980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f973800a7f0 tx=0x7f97280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecff98794cf7d0b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380072d0 con 0x7f97280052a0 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973802f070 con 0x7f97280052a0 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.440+0000 7f973effd640 1 -- 172.21.15.124:0/1968430469 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738007670 con 0x7f97280052a0 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 -- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 msgr2=0x7f972800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 0x7f972800b980 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f973800a7f0 tx=0x7f97280057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 -- 172.21.15.124:0/1968430469 shutdown_connections 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 --2- 172.21.15.124:0/1968430469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97280052a0 0x7f972800b980 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 -- 172.21.15.124:0/1968430469 >> 172.21.15.124:0/1968430469 conn(0x7f97400b2c10 msgr2=0x7f97400b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 -- 172.21.15.124:0/1968430469 shutdown_connections 2023-12-13T23:55:25.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.441+0000 7f9746961640 1 -- 172.21.15.124:0/1968430469 wait complete. 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.442+0000 7f9746961640 1 Processor -- start 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.442+0000 7f9746961640 1 -- start start 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97280052a0 0x7f9740149c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9740149450 0x7f974015d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97401f5160 con 0x7f974015d550 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97401498d0 con 0x7f9740149450 2023-12-13T23:55:25.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9746961640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974015d9d0 con 0x7f97280052a0 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57546/0 (socket says 172.21.15.124:57546) 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 -- 172.21.15.124:0/877824514 learned_addr learned my addr 172.21.15.124:0/877824514 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f973ffff640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97280052a0 0x7f9740149c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f973f7fe640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9740149450 0x7f974015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 -- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97280052a0 msgr2=0x7f9740149c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97280052a0 0x7f9740149c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 -- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9740149450 msgr2=0x7f974015d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9740149450 0x7f974015d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.443+0000 7f9744efe640 1 -- 172.21.15.124:0/877824514 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97400b5170 con 0x7f974015d550 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f9744efe640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f973800a2c0 tx=0x7f974015b640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380019b0 con 0x7f974015d550 2023-12-13T23:55:25.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9738001b10 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97401f4570 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97380072d0 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.444+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97401f47c0 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.445+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9740069400 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.445+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9738031030 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.446+0000 7f973cff9640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 0x7f972c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.446+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97380aac70 con 0x7f974015d550 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.446+0000 7f973ffff640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 0x7f972c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.446+0000 7f973ffff640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 0x7f972c06d680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9730000c00 tx=0x7f9730009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.450+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9738075a30 con 0x7f974015d550 2023-12-13T23:55:25.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.617+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9740069400 con 0x7f974015d550 2023-12-13T23:55:25.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.617+0000 7f973cff9640 1 -- 172.21.15.124:0/877824514 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9738044020 con 0x7f974015d550 2023-12-13T23:55:25.616 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 msgr2=0x7f972c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 0x7f972c06d680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9730000c00 tx=0x7f9730009040 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 msgr2=0x7f97400b3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f973800a2c0 tx=0x7f974015b640 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 shutdown_connections 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f972c06b1c0 0x7f972c06d680 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9740149450 0x7f974015d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f974015d550 0x7f97400b3d00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 --2- 172.21.15.124:0/877824514 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97280052a0 0x7f9740149c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 >> 172.21.15.124:0/877824514 conn(0x7f97400b2c10 msgr2=0x7f97400b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 shutdown_connections 2023-12-13T23:55:25.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.619+0000 7f9746961640 1 -- 172.21.15.124:0/877824514 wait complete. 2023-12-13T23:55:25.629 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:25.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.735+0000 7f2579090640 1 Processor -- start 2023-12-13T23:55:25.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.735+0000 7f2579090640 1 -- start start 2023-12-13T23:55:25.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2579090640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f257415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2579090640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2574071770 con 0x7f25740ceb70 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2579090640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25740ccb60 con 0x7f25740b4370 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2579090640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25740ccdc0 con 0x7f2574144980 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f257259c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f25740ceb70 0x7f25740cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43730/0 (socket says 172.21.15.124:43730) 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f257259c640 1 -- 172.21.15.124:0/2669076021 learned_addr learned my addr 172.21.15.124:0/2669076021 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854465340 0 0) 0x7f2574071770 con 0x7f25740ceb70 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2572d9d640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f257415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f255c003520 con 0x7f25740ceb70 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2920644358 0 0) 0x7f255c003520 con 0x7f25740ceb70 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 msgr2=0x7f257415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f257415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 >> v1:172.21.15.203:6789/0 conn(0x7f25740b4370 legacy=0x7f2574071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740cd0a0 con 0x7f25740ceb70 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f25680019f0 con 0x7f25740ceb70 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 >> v1:172.21.15.124:6789/0 conn(0x7f25740ceb70 legacy=0x7f25740cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f255c0039a0 0x7f255c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f255c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c004d00 0x7f255c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740ccdc0 con 0x7f255c0039a0 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f255c00b980 con 0x7f255c004d00 2023-12-13T23:55:25.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.736+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f255c00bb40 con 0x7f2574144980 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f255c0039a0 0x7f255c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f257259c640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f255c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f257359e640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c004d00 0x7f255c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 -- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 msgr2=0x7f255c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2574144980 0x7f255c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 -- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c004d00 msgr2=0x7f255c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c004d00 0x7f255c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 -- 172.21.15.124:0/2669076021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255c003520 con 0x7f255c0039a0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.737+0000 7f2572d9d640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f255c0039a0 0x7f255c003d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f256c009ac0 tx=0x7f255c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614df987073be214 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c02c600 con 0x7f255c0039a0 2023-12-13T23:55:25.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f256c02c760 con 0x7f255c0039a0 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2571d9b640 1 -- 172.21.15.124:0/2669076021 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f256c02caf0 con 0x7f255c0039a0 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 -- 172.21.15.124:0/2669076021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f255c0039a0 msgr2=0x7f255c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 --2- 172.21.15.124:0/2669076021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f255c0039a0 0x7f255c003d80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f256c009ac0 tx=0x7f255c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f257359e640 1 -- 172.21.15.124:0/2669076021 reap_dead start 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 -- 172.21.15.124:0/2669076021 shutdown_connections 2023-12-13T23:55:25.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 -- 172.21.15.124:0/2669076021 >> 172.21.15.124:0/2669076021 conn(0x7f257406a4d0 msgr2=0x7f257406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 -- 172.21.15.124:0/2669076021 shutdown_connections 2023-12-13T23:55:25.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.738+0000 7f2579090640 1 -- 172.21.15.124:0/2669076021 wait complete. 2023-12-13T23:55:25.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 Processor -- start 2023-12-13T23:55:25.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 -- start start 2023-12-13T23:55:25.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c0039a0 0x7f25741365a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f257415ad80 0x7f257415b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25741f3b20 con 0x7f25740ab380 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25740ae1e0 con 0x7f255c0039a0 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f2579090640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25740adcd0 con 0x7f257415ad80 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f257259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f257259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57568/0 (socket says 172.21.15.124:57568) 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.740+0000 7f257259c640 1 -- 172.21.15.124:0/1023624802 learned_addr learned my addr 172.21.15.124:0/1023624802 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 -- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f257415ad80 msgr2=0x7f257415b200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f257415ad80 0x7f257415b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f2572d9d640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c0039a0 0x7f25741365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 -- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c0039a0 msgr2=0x7f25741365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c0039a0 0x7f25741365a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 -- 172.21.15.124:0/1023624802 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25740ab800 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f257259c640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2568007a80 tx=0x7f256800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568013420 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25680135a0 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25740adf00 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2568013950 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.741+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f257415a580 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.742+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2574155b10 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.742+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2568013ab0 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.743+0000 7f255b7fe640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 0x7f256006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.743+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f256808a220 con 0x7f25740ab380 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.743+0000 7f2572d9d640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 0x7f256006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:25.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.743+0000 7f2572d9d640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 0x7f256006d6b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f256c036000 tx=0x7f256c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:25.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.747+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2568054fe0 con 0x7f25740ab380 2023-12-13T23:55:25.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.916+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2574069400 con 0x7f25740ab380 2023-12-13T23:55:25.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.916+0000 7f255b7fe640 1 -- 172.21.15.124:0/1023624802 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2568058600 con 0x7f25740ab380 2023-12-13T23:55:25.915 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 msgr2=0x7f256006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 0x7f256006d6b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f256c036000 tx=0x7f256c030000 comp rx=0 tx=0).stop 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 msgr2=0x7f25740ad790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2568007a80 tx=0x7f256800e530 comp rx=0 tx=0).stop 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 shutdown_connections 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f256006b1f0 0x7f256006d6b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f257415ad80 0x7f257415b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25740ab380 0x7f25740ad790 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 --2- 172.21.15.124:0/1023624802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f255c0039a0 0x7f25741365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:25.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 >> 172.21.15.124:0/1023624802 conn(0x7f257406a4d0 msgr2=0x7f2574157380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:25.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 shutdown_connections 2023-12-13T23:55:25.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:25.918+0000 7f2579090640 1 -- 172.21.15.124:0/1023624802 wait complete. 2023-12-13T23:55:25.928 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:26.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.032+0000 7f5eb0368640 1 Processor -- start 2023-12-13T23:55:26.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.032+0000 7f5eb0368640 1 -- start start 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5eb0368640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5ea80d2910 0x7f5ea80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5eb0368640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5ea80d3730 0x7f5ea80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5eb0368640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea80b2b10 con 0x7f5ea815d330 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5eb0368640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80d3bc0 con 0x7f5ea80d2910 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5eb0368640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea80d1590 con 0x7f5ea80d3730 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5ead903640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5ea80d2910 0x7f5ea80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5ead903640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5ea80d2910 0x7f5ea80d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57582/0 (socket says 172.21.15.124:57582) 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.033+0000 7f5ead903640 1 -- 172.21.15.124:0/2440147493 learned_addr learned my addr 172.21.15.124:0/2440147493 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead903640 1 -- 172.21.15.124:0/2440147493 >> v2:172.21.15.203:3300/0 conn(0x7f5ea80d3730 msgr2=0x7f5ea80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:26.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5eae905640 1 --2- 172.21.15.124:0/2440147493 >> v2:172.21.15.203:3300/0 conn(0x7f5ea80d3730 0x7f5ea80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead903640 1 --2- 172.21.15.124:0/2440147493 >> v2:172.21.15.203:3300/0 conn(0x7f5ea80d3730 0x7f5ea80b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead903640 1 -- 172.21.15.124:0/2440147493 >> v1:172.21.15.203:6789/0 conn(0x7f5ea815d330 legacy=0x7f5ea80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:26.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead903640 1 -- 172.21.15.124:0/2440147493 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80d1870 con 0x7f5ea80d2910 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead903640 1 --2- 172.21.15.124:0/2440147493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea80d2910 0x7f5ea80d31f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ea806c500 tx=0x7f5e9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44efc0799ccd1cee server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead102640 1 -- 172.21.15.124:0/2440147493 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e980317e0 con 0x7f5ea80d2910 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead102640 1 -- 172.21.15.124:0/2440147493 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e98031940 con 0x7f5ea80d2910 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.034+0000 7f5ead102640 1 -- 172.21.15.124:0/2440147493 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98031cd0 con 0x7f5ea80d2910 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 -- 172.21.15.124:0/2440147493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea80d2910 msgr2=0x7f5ea80d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 --2- 172.21.15.124:0/2440147493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea80d2910 0x7f5ea80d31f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5ea806c500 tx=0x7f5e9802b880 comp rx=0 tx=0).stop 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 -- 172.21.15.124:0/2440147493 shutdown_connections 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 --2- 172.21.15.124:0/2440147493 >> v2:172.21.15.203:3300/0 conn(0x7f5ea80d3730 0x7f5ea80b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 --2- 172.21.15.124:0/2440147493 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea80d2910 0x7f5ea80d31f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 -- 172.21.15.124:0/2440147493 >> 172.21.15.124:0/2440147493 conn(0x7f5ea8072020 msgr2=0x7f5ea8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 -- 172.21.15.124:0/2440147493 shutdown_connections 2023-12-13T23:55:26.035 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.035+0000 7f5eb0368640 1 -- 172.21.15.124:0/2440147493 wait complete. 2023-12-13T23:55:26.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.036+0000 7f5eb0368640 1 Processor -- start 2023-12-13T23:55:26.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.036+0000 7f5eb0368640 1 -- start start 2023-12-13T23:55:26.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ea80d2910 0x7f5ea814f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea80d3730 0x7f5ea81513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80c7bf0 con 0x7f5ea815d330 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80c7e40 con 0x7f5ea80d3730 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eb0368640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80c8120 con 0x7f5ea80d2910 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57586/0 (socket says 172.21.15.124:57586) 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 -- 172.21.15.124:0/826462570 learned_addr learned my addr 172.21.15.124:0/826462570 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae104640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ea80d2910 0x7f5ea814f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5ead903640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea80d3730 0x7f5ea81513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 -- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ea80d2910 msgr2=0x7f5ea814f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ea80d2910 0x7f5ea814f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 -- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea80d3730 msgr2=0x7f5ea81513d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea80d3730 0x7f5ea81513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.037+0000 7f5eae905640 1 -- 172.21.15.124:0/826462570 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e980094d0 con 0x7f5ea815d330 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5eae905640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c7f0 tx=0x7f5ea000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0004220 con 0x7f5ea815d330 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0004380 con 0x7f5ea815d330 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea80c8400 con 0x7f5ea815d330 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0007860 con 0x7f5ea815d330 2023-12-13T23:55:26.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.038+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea814fe20 con 0x7f5ea815d330 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.039+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea80c8920 con 0x7f5ea815d330 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.039+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5ea0007bd0 con 0x7f5ea815d330 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.040+0000 7f5e9effd640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 0x7f5e8c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.040+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ea0089980 con 0x7f5ea815d330 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.040+0000 7f5eae104640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 0x7f5e8c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.040+0000 7f5eae104640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 0x7f5e8c06d680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4007dc0 tx=0x7f5ea40072f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.044+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5ea80c8920 con 0x7f5ea815d330 2023-12-13T23:55:26.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.211+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ea80c8920 con 0x7f5ea815d330 2023-12-13T23:55:26.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.211+0000 7f5e9effd640 1 -- 172.21.15.124:0/826462570 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ea80c8920 con 0x7f5ea815d330 2023-12-13T23:55:26.210 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 msgr2=0x7f5e8c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 0x7f5e8c06d680 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4007dc0 tx=0x7f5ea40072f0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 msgr2=0x7f5ea8151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c7f0 tx=0x7f5ea000ccb0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 shutdown_connections 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8c06b1c0 0x7f5e8c06d680 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5ea80d3730 0x7f5ea81513d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ea815d330 0x7f5ea8151910 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 --2- 172.21.15.124:0/826462570 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5ea80d2910 0x7f5ea814f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 >> 172.21.15.124:0/826462570 conn(0x7f5ea8072020 msgr2=0x7f5ea8072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 shutdown_connections 2023-12-13T23:55:26.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.213+0000 7f5eb0368640 1 -- 172.21.15.124:0/826462570 wait complete. 2023-12-13T23:55:26.222 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:26.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.324+0000 7f632869b640 1 Processor -- start 2023-12-13T23:55:26.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.324+0000 7f632869b640 1 -- start start 2023-12-13T23:55:26.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f632869b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6320142270 0x7f63201426c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f632869b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6320161d20 con 0x7f6320160fb0 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f632869b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632015e760 con 0x7f632006ea50 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f632869b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63200d1ee0 con 0x7f6320142270 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6326c38640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6320142270 0x7f63201426c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6325c36640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6320160fb0 0x7f6320161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43740/0 (socket says 172.21.15.124:43740) 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6325c36640 1 -- 172.21.15.124:0/2672794406 learned_addr learned my addr 172.21.15.124:0/2672794406 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6326c38640 1 -- 172.21.15.124:0/2672794406 >> v1:172.21.15.203:6789/0 conn(0x7f632006ea50 legacy=0x7f632006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6326c38640 1 -- 172.21.15.124:0/2672794406 >> v1:172.21.15.124:6789/0 conn(0x7f6320160fb0 legacy=0x7f6320161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.325+0000 7f6326c38640 1 -- 172.21.15.124:0/2672794406 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632015d9a0 con 0x7f6320142270 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f6326c38640 1 --2- 172.21.15.124:0/2672794406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f63201426c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6318009d90 tx=0x7f631802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f743b4cf1f1dcc3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f6325435640 1 -- 172.21.15.124:0/2672794406 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f631802cc60 con 0x7f6320142270 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f6325435640 1 -- 172.21.15.124:0/2672794406 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f631802cdc0 con 0x7f6320142270 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f6325435640 1 -- 172.21.15.124:0/2672794406 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6318034660 con 0x7f6320142270 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f632869b640 1 -- 172.21.15.124:0/2672794406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 msgr2=0x7f63201426c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.326+0000 7f632869b640 1 --2- 172.21.15.124:0/2672794406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f63201426c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f6318009d90 tx=0x7f631802beb0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.327+0000 7f632869b640 1 -- 172.21.15.124:0/2672794406 shutdown_connections 2023-12-13T23:55:26.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.327+0000 7f632869b640 1 --2- 172.21.15.124:0/2672794406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f63201426c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.327+0000 7f632869b640 1 -- 172.21.15.124:0/2672794406 >> 172.21.15.124:0/2672794406 conn(0x7f632006a6b0 msgr2=0x7f632006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.327+0000 7f632869b640 1 -- 172.21.15.124:0/2672794406 shutdown_connections 2023-12-13T23:55:26.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.327+0000 7f632869b640 1 -- 172.21.15.124:0/2672794406 wait complete. 2023-12-13T23:55:26.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 Processor -- start 2023-12-13T23:55:26.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 -- start start 2023-12-13T23:55:26.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f632006ea50 0x7f6320155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6320160fb0 0x7f6320141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63201562b0 con 0x7f6320142270 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632013f1d0 con 0x7f6320160fb0 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.328+0000 7f632869b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f632013eef0 con 0x7f632006ea50 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57600/0 (socket says 172.21.15.124:57600) 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 -- 172.21.15.124:0/1563879072 learned_addr learned my addr 172.21.15.124:0/1563879072 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6326437640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f632006ea50 0x7f6320155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6326c38640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6320160fb0 0x7f6320141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 -- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f632006ea50 msgr2=0x7f6320155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f632006ea50 0x7f6320155d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 -- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6320160fb0 msgr2=0x7f6320141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6320160fb0 0x7f6320141230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 -- 172.21.15.124:0/1563879072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318009a70 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6325c36640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f631000c4a0 tx=0x7f631000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310005ce0 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6310005e40 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f632013f4f0 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6310007270 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.329+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f632013fab0 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.330+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6320069400 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.330+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6310007490 con 0x7f6320142270 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.331+0000 7f6316ffd640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 0x7f630406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.331+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6310088dd0 con 0x7f6320142270 2023-12-13T23:55:26.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.331+0000 7f6326437640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 0x7f630406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.332+0000 7f6326437640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 0x7f630406d680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f631c001500 tx=0x7f631c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.335+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6310053c40 con 0x7f6320142270 2023-12-13T23:55:26.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.535+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63201e7620 con 0x7f6320142270 2023-12-13T23:55:26.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.536+0000 7f6316ffd640 1 -- 172.21.15.124:0/1563879072 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f631008a090 con 0x7f6320142270 2023-12-13T23:55:26.534 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:26.534 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:26.535 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.537+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 msgr2=0x7f630406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.537+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 0x7f630406d680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f631c001500 tx=0x7f631c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.537+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 msgr2=0x7f632015a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.537+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f631000c4a0 tx=0x7f631000c960 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 shutdown_connections 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f630406b1c0 0x7f630406d680 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6320160fb0 0x7f6320141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6320142270 0x7f632015a990 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 --2- 172.21.15.124:0/1563879072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f632006ea50 0x7f6320155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 >> 172.21.15.124:0/1563879072 conn(0x7f632006a6b0 msgr2=0x7f632006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 shutdown_connections 2023-12-13T23:55:26.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.538+0000 7f632869b640 1 -- 172.21.15.124:0/1563879072 wait complete. 2023-12-13T23:55:26.547 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:26.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.647+0000 7f3a15b1d640 1 Processor -- start 2023-12-13T23:55:26.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.647+0000 7f3a15b1d640 1 -- start start 2023-12-13T23:55:26.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a15b1d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a10143490 0x7f3a100b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a15b1d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a1015abf0 0x7f3a100b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a15b1d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1015e2a0 con 0x7f3a1015e730 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a15b1d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1015b520 con 0x7f3a10143490 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a15b1d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a10143870 con 0x7f3a1015abf0 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a10143490 0x7f3a100b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0effd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3a1015e730 0x7f3a1015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43756/0 (socket says 172.21.15.124:43756) 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3a10143490 0x7f3a100b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57616/0 (socket says 172.21.15.124:57616) 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0effd640 1 -- 172.21.15.124:0/4087291495 learned_addr learned my addr 172.21.15.124:0/4087291495 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0ffff640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a1015abf0 0x7f3a100b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 -- 172.21.15.124:0/4087291495 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a1015abf0 msgr2=0x7f3a100b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a1015abf0 0x7f3a100b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0e7fc640 1 -- 172.21.15.124:0/4087291495 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789102770 0 0) 0x7f3a1015e2a0 con 0x7f3a1015e730 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 -- 172.21.15.124:0/4087291495 >> v1:172.21.15.124:6789/0 conn(0x7f3a1015e730 legacy=0x7f3a1015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.648+0000 7f3a0f7fe640 1 -- 172.21.15.124:0/4087291495 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1015b240 con 0x7f3a10143490 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.649+0000 7f3a0f7fe640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a10143490 0x7f3a100b0cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3a000097a0 tx=0x7f3a0002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf732774647d6a75 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.649+0000 7f3a0e7fc640 1 -- 172.21.15.124:0/4087291495 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00031830 con 0x7f3a10143490 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.649+0000 7f3a0e7fc640 1 -- 172.21.15.124:0/4087291495 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a00031990 con 0x7f3a10143490 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.649+0000 7f3a0e7fc640 1 -- 172.21.15.124:0/4087291495 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a00031d20 con 0x7f3a10143490 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4087291495 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a10143490 msgr2=0x7f3a100b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a10143490 0x7f3a100b0cf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3a000097a0 tx=0x7f3a0002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4087291495 shutdown_connections 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a1015abf0 0x7f3a100b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4087291495 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a10143490 0x7f3a100b0cf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4087291495 >> 172.21.15.124:0/4087291495 conn(0x7f3a1006a4d0 msgr2=0x7f3a1006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4087291495 shutdown_connections 2023-12-13T23:55:26.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.650+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4087291495 wait complete. 2023-12-13T23:55:26.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.651+0000 7f3a15b1d640 1 Processor -- start 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.651+0000 7f3a15b1d640 1 -- start start 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a10143490 0x7f3a101416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 0x7f3a1013edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100aede0 con 0x7f3a1015abf0 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100af0c0 con 0x7f3a1015e730 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100af3a0 con 0x7f3a10143490 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57620/0 (socket says 172.21.15.124:57620) 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 -- 172.21.15.124:0/4072054675 learned_addr learned my addr 172.21.15.124:0/4072054675 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0f7fe640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a10143490 0x7f3a101416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0ffff640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 0x7f3a1013edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 -- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a10143490 msgr2=0x7f3a101416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a10143490 0x7f3a101416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 -- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 msgr2=0x7f3a1013edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 0x7f3a1013edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 -- 172.21.15.124:0/4072054675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00009400 con 0x7f3a1015abf0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0ffff640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 0x7f3a1013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a0effd640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c430 tx=0x7f3a0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a040074e0 con 0x7f3a1015abf0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a04007660 con 0x7f3a1015abf0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a04004a20 con 0x7f3a1015abf0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a100af680 con 0x7f3a1015abf0 2023-12-13T23:55:26.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.652+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a101ef3f0 con 0x7f3a1015abf0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.653+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a100afba0 con 0x7f3a1015abf0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.654+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3a04004b80 con 0x7f3a1015abf0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.654+0000 7f39f7fff640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 0x7f39fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.654+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3a04089720 con 0x7f3a1015abf0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.655+0000 7f3a0f7fe640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 0x7f39fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.655+0000 7f3a0f7fe640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 0x7f39fc06d680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a00036000 tx=0x7f3a0003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.659+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3a100afba0 con 0x7f3a1015abf0 2023-12-13T23:55:26.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.826+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a100afba0 con 0x7f3a1015abf0 2023-12-13T23:55:26.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.827+0000 7f39f7fff640 1 -- 172.21.15.124:0/4072054675 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f3a100afba0 con 0x7f3a1015abf0 2023-12-13T23:55:26.825 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:26.825 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.828+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 msgr2=0x7f39fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.828+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 0x7f39fc06d680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f3a00036000 tx=0x7f3a0003c040 comp rx=0 tx=0).stop 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.828+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 msgr2=0x7f3a101eeeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.828+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3a0400c430 tx=0x7f3a0400c8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 shutdown_connections 2023-12-13T23:55:26.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f39fc06b1c0 0x7f39fc06d680 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3a1015e730 0x7f3a1013edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3a1015abf0 0x7f3a101eeeb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 --2- 172.21.15.124:0/4072054675 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3a10143490 0x7f3a101416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 >> 172.21.15.124:0/4072054675 conn(0x7f3a1006a4d0 msgr2=0x7f3a10071ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 shutdown_connections 2023-12-13T23:55:26.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.829+0000 7f3a15b1d640 1 -- 172.21.15.124:0/4072054675 wait complete. 2023-12-13T23:55:26.838 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:26.838 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:26.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.942+0000 7f63d47ec640 1 Processor -- start 2023-12-13T23:55:26.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.942+0000 7f63d47ec640 1 -- start start 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d47ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc143490 0x7f63cc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d47ec640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63cc15e730 0x7f63cc0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d47ec640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc15e2a0 con 0x7f63cc15a4e0 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d47ec640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc15b520 con 0x7f63cc15e730 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d47ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc143870 con 0x7f63cc143490 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d1d87640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63cc15e730 0x7f63cc0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d1d87640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63cc15e730 0x7f63cc0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57624/0 (socket says 172.21.15.124:57624) 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d1d87640 1 -- 172.21.15.124:0/1646662131 learned_addr learned my addr 172.21.15.124:0/1646662131 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d2588640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc143490 0x7f63cc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.943+0000 7f63d1d87640 1 -- 172.21.15.124:0/1646662131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc143490 msgr2=0x7f63cc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1d87640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc143490 0x7f63cc0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1d87640 1 -- 172.21.15.124:0/1646662131 >> v1:172.21.15.203:6789/0 conn(0x7f63cc15a4e0 legacy=0x7f63cc15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:26.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1d87640 1 -- 172.21.15.124:0/1646662131 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc15b240 con 0x7f63cc15e730 2023-12-13T23:55:26.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1d87640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc15e730 0x7f63cc0b1230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6aa6324de0014ca server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1586640 1 -- 172.21.15.124:0/1646662131 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc031640 con 0x7f63cc15e730 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1586640 1 -- 172.21.15.124:0/1646662131 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0317a0 con 0x7f63cc15e730 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.944+0000 7f63d1586640 1 -- 172.21.15.124:0/1646662131 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc031b30 con 0x7f63cc15e730 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 -- 172.21.15.124:0/1646662131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc15e730 msgr2=0x7f63cc0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc15e730 0x7f63cc0b1230 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f63bc009870 tx=0x7f63bc02b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 -- 172.21.15.124:0/1646662131 shutdown_connections 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc15e730 0x7f63cc0b1230 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1646662131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc143490 0x7f63cc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 -- 172.21.15.124:0/1646662131 >> 172.21.15.124:0/1646662131 conn(0x7f63cc06a4d0 msgr2=0x7f63cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 -- 172.21.15.124:0/1646662131 shutdown_connections 2023-12-13T23:55:26.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.945+0000 7f63d47ec640 1 -- 172.21.15.124:0/1646662131 wait complete. 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.946+0000 7f63d47ec640 1 Processor -- start 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.946+0000 7f63d47ec640 1 -- start start 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63cc15a4e0 0x7f63cc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc15e730 0x7f63cc06c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0abf00 con 0x7f63cc143490 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0ac1e0 con 0x7f63cc15a4e0 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d47ec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0ac4c0 con 0x7f63cc15e730 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57634/0 (socket says 172.21.15.124:57634) 2023-12-13T23:55:26.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 -- 172.21.15.124:0/1088737875 learned_addr learned my addr 172.21.15.124:0/1088737875 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 -- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc15e730 msgr2=0x7f63cc06c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc15e730 0x7f63cc06c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 -- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63cc15a4e0 msgr2=0x7f63cc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63cc15a4e0 0x7f63cc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 -- 172.21.15.124:0/1088737875 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0094d0 con 0x7f63cc143490 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63d2588640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63c800ce10 tx=0x7f63c800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8003b90 con 0x7f63cc143490 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.947+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c8003cf0 con 0x7f63cc143490 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.948+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8003ea0 con 0x7f63cc143490 2023-12-13T23:55:26.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.948+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0ac7a0 con 0x7f63cc143490 2023-12-13T23:55:26.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.948+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc13e6c0 con 0x7f63cc143490 2023-12-13T23:55:26.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.948+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc13e850 con 0x7f63cc143490 2023-12-13T23:55:26.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.949+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f63c800f9c0 con 0x7f63cc143490 2023-12-13T23:55:26.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.949+0000 7f63c2ffd640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 0x7f63b006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:26.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.949+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63c808c1b0 con 0x7f63cc143490 2023-12-13T23:55:26.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.950+0000 7f63d1d87640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 0x7f63b006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:26.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.950+0000 7f63d1d87640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 0x7f63b006d6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:26.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:26.953+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f63c8056f70 con 0x7f63cc143490 2023-12-13T23:55:27.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.119+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63cc0accc0 con 0x7f63cc143490 2023-12-13T23:55:27.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.119+0000 7f63c2ffd640 1 -- 172.21.15.124:0/1088737875 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63c805a590 con 0x7f63cc143490 2023-12-13T23:55:27.118 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 msgr2=0x7f63b006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 0x7f63b006d6b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f63bc036000 tx=0x7f63bc030000 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 msgr2=0x7f63cc13e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f63c800ce10 tx=0x7f63c800a660 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 shutdown_connections 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63b006b1f0 0x7f63b006d6b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63cc15a4e0 0x7f63cc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63cc15e730 0x7f63cc06c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 --2- 172.21.15.124:0/1088737875 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63cc143490 0x7f63cc13e180 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 >> 172.21.15.124:0/1088737875 conn(0x7f63cc06a4d0 msgr2=0x7f63cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 shutdown_connections 2023-12-13T23:55:27.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.121+0000 7f63d47ec640 1 -- 172.21.15.124:0/1088737875 wait complete. 2023-12-13T23:55:27.130 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:27.232 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.233+0000 7f85ddec9640 1 Processor -- start 2023-12-13T23:55:27.232 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.233+0000 7f85ddec9640 1 -- start start 2023-12-13T23:55:27.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85ddec9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f85d815d330 0x7f85d80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85ddec9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d80d2910 0x7f85d80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85ddec9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d80b2b10 con 0x7f85d80d3730 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85ddec9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d80d3b50 con 0x7f85d815d330 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85ddec9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d80d1590 con 0x7f85d80d2910 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85d77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f85d815d330 0x7f85d80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85d77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f85d815d330 0x7f85d80b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57650/0 (socket says 172.21.15.124:57650) 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85d77fe640 1 -- 172.21.15.124:0/4012573767 learned_addr learned my addr 172.21.15.124:0/4012573767 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.234+0000 7f85d6ffd640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d80d2910 0x7f85d80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d77fe640 1 -- 172.21.15.124:0/4012573767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d80d2910 msgr2=0x7f85d80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d80d2910 0x7f85d80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d77fe640 1 -- 172.21.15.124:0/4012573767 >> v1:172.21.15.203:6789/0 conn(0x7f85d80d3730 legacy=0x7f85d80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d77fe640 1 -- 172.21.15.124:0/4012573767 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d80d1870 con 0x7f85d815d330 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d815d330 0x7f85d80b8b80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e50 tx=0x7f85c802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412cede3d4b977b6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d67fc640 1 -- 172.21.15.124:0/4012573767 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c802c9e0 con 0x7f85d815d330 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d67fc640 1 -- 172.21.15.124:0/4012573767 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c802cb40 con 0x7f85d815d330 2023-12-13T23:55:27.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.235+0000 7f85d67fc640 1 -- 172.21.15.124:0/4012573767 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80343f0 con 0x7f85d815d330 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 -- 172.21.15.124:0/4012573767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d815d330 msgr2=0x7f85d80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d815d330 0x7f85d80b8b80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85c8009e50 tx=0x7f85c802be40 comp rx=0 tx=0).stop 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 -- 172.21.15.124:0/4012573767 shutdown_connections 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d80d2910 0x7f85d80d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4012573767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d815d330 0x7f85d80b8b80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 -- 172.21.15.124:0/4012573767 >> 172.21.15.124:0/4012573767 conn(0x7f85d8072020 msgr2=0x7f85d8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 -- 172.21.15.124:0/4012573767 shutdown_connections 2023-12-13T23:55:27.236 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.236+0000 7f85ddec9640 1 -- 172.21.15.124:0/4012573767 wait complete. 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.237+0000 7f85ddec9640 1 Processor -- start 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.237+0000 7f85ddec9640 1 -- start start 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 0x7f85d80b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d815d330 0x7f85d8160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81f6e00 con 0x7f85d80d3730 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81609a0 con 0x7f85d80d2910 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85ddec9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d81462e0 con 0x7f85d815d330 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57664/0 (socket says 172.21.15.124:57664) 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 -- 172.21.15.124:0/4199808066 learned_addr learned my addr 172.21.15.124:0/4199808066 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d7fff640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d815d330 0x7f85d8160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 0x7f85d80b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 -- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d815d330 msgr2=0x7f85d8160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d815d330 0x7f85d8160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 -- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 msgr2=0x7f85d80b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 0x7f85d80b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d6ffd640 1 -- 172.21.15.124:0/4199808066 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c8009b30 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.238+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 0x7f85d80b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85d6ffd640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d620 tx=0x7f85cc00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00b620 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85cc00b780 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d81f7220 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85cc00bb30 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.239+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d81f6740 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.240+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d8069400 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.240+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f85cc00bc90 con 0x7f85d80d3730 2023-12-13T23:55:27.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.240+0000 7f85bffff640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 0x7f85c406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.241+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f85cc089d70 con 0x7f85d80d3730 2023-12-13T23:55:27.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.241+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 0x7f85c406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.241+0000 7f85d77fe640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 0x7f85c406d680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85c8035000 tx=0x7f85c8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.245+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f85cc054b30 con 0x7f85d80d3730 2023-12-13T23:55:27.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.413+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85d81f6ae0 con 0x7f85d80d3730 2023-12-13T23:55:27.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.414+0000 7f85bffff640 1 -- 172.21.15.124:0/4199808066 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85cc058150 con 0x7f85d80d3730 2023-12-13T23:55:27.412 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:27.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 msgr2=0x7f85c406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 0x7f85c406d680 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f85c8035000 tx=0x7f85c8030000 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 msgr2=0x7f85d815ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00d620 tx=0x7f85cc00dae0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 shutdown_connections 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f85c406b1c0 0x7f85c406d680 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f85d815d330 0x7f85d8160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f85d80d3730 0x7f85d815ff20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 --2- 172.21.15.124:0/4199808066 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f85d80d2910 0x7f85d80b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 >> 172.21.15.124:0/4199808066 conn(0x7f85d8072020 msgr2=0x7f85d80b2150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 shutdown_connections 2023-12-13T23:55:27.415 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.416+0000 7f85ddec9640 1 -- 172.21.15.124:0/4199808066 wait complete. 2023-12-13T23:55:27.425 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:27.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.529+0000 7f23a109b640 1 Processor -- start 2023-12-13T23:55:27.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 -- start start 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 0x7f239c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f239c0d2910 0x7f239c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f239c0b2b10 con 0x7f239c0d2d90 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f239c0d1590 con 0x7f239c0d2910 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f23a109b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0d1870 con 0x7f239c15d330 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.530+0000 7f2393fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f239c0d2910 0x7f239c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f239c0d2910 0x7f239c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57680/0 (socket says 172.21.15.124:57680) 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 -- 172.21.15.124:0/3249844808 learned_addr learned my addr 172.21.15.124:0/3249844808 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f239ad9d640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 0x7f239c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 -- 172.21.15.124:0/3249844808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 msgr2=0x7f239c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 0x7f239c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 -- 172.21.15.124:0/3249844808 >> v1:172.21.15.203:6789/0 conn(0x7f239c0d2d90 legacy=0x7f239c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 -- 172.21.15.124:0/3249844808 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c0d1b50 con 0x7f239c0d2910 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f239ad9d640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 0x7f239c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.531+0000 7f2393fff640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2910 0x7f239c0d31f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2384009870 tx=0x7f238402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb86e1973a71906f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f239a59c640 1 -- 172.21.15.124:0/3249844808 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840317c0 con 0x7f239c0d2910 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f239a59c640 1 -- 172.21.15.124:0/3249844808 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384031920 con 0x7f239c0d2910 2023-12-13T23:55:27.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f239a59c640 1 -- 172.21.15.124:0/3249844808 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384031cb0 con 0x7f239c0d2910 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 -- 172.21.15.124:0/3249844808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2910 msgr2=0x7f239c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2910 0x7f239c0d31f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2384009870 tx=0x7f238402b860 comp rx=0 tx=0).stop 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 -- 172.21.15.124:0/3249844808 shutdown_connections 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2910 0x7f239c0d31f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 --2- 172.21.15.124:0/3249844808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c15d330 0x7f239c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 -- 172.21.15.124:0/3249844808 >> 172.21.15.124:0/3249844808 conn(0x7f239c072020 msgr2=0x7f239c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 -- 172.21.15.124:0/3249844808 shutdown_connections 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.532+0000 7f23a109b640 1 -- 172.21.15.124:0/3249844808 wait complete. 2023-12-13T23:55:27.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 Processor -- start 2023-12-13T23:55:27.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 -- start start 2023-12-13T23:55:27.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c0d2910 0x7f239c14cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f239c15d330 0x7f239c0caec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c14d0b0 con 0x7f239c0d2d90 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c157640 con 0x7f239c15d330 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f23a109b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c0c7cd0 con 0x7f239c0d2910 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f2393fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f2393fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57686/0 (socket says 172.21.15.124:57686) 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.534+0000 7f2393fff640 1 -- 172.21.15.124:0/451561292 learned_addr learned my addr 172.21.15.124:0/451561292 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f239ad9d640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c0d2910 0x7f239c14cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f239b59e640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f239c15d330 0x7f239c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 -- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c0d2910 msgr2=0x7f239c14cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c0d2910 0x7f239c14cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 -- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f239c15d330 msgr2=0x7f239c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f239c15d330 0x7f239c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 -- 172.21.15.124:0/451561292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23840094d0 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f2393fff640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2384000c00 tx=0x7f23840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840329a0 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2384032b20 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c0c7fb0 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23840453f0 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.535+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c0c8570 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.536+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c069400 con 0x7f239c0d2d90 2023-12-13T23:55:27.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.536+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f238404d3f0 con 0x7f239c0d2d90 2023-12-13T23:55:27.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.537+0000 7f23937fe640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 0x7f237c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.537+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23840ab920 con 0x7f239c0d2d90 2023-12-13T23:55:27.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.537+0000 7f239ad9d640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 0x7f237c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.537+0000 7f239ad9d640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 0x7f237c06d6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a990 tx=0x7f238c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.541+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f23840766e0 con 0x7f239c0d2d90 2023-12-13T23:55:27.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.707+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f239c0d1060 con 0x7f239c0d2d90 2023-12-13T23:55:27.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.708+0000 7f23937fe640 1 -- 172.21.15.124:0/451561292 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2384079d00 con 0x7f239c0d2d90 2023-12-13T23:55:27.706 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:27.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.709+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 msgr2=0x7f237c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.709+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 0x7f237c06d6b0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f238c00a990 tx=0x7f238c008040 comp rx=0 tx=0).stop 2023-12-13T23:55:27.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.709+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 msgr2=0x7f239c157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.709+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2384000c00 tx=0x7f23840046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 shutdown_connections 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f237c06b1f0 0x7f237c06d6b0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f239c15d330 0x7f239c0caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f239c0d2d90 0x7f239c157100 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 --2- 172.21.15.124:0/451561292 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f239c0d2910 0x7f239c14cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 >> 172.21.15.124:0/451561292 conn(0x7f239c072020 msgr2=0x7f239c0cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 shutdown_connections 2023-12-13T23:55:27.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.710+0000 7f23a109b640 1 -- 172.21.15.124:0/451561292 wait complete. 2023-12-13T23:55:27.719 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:27.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.821+0000 7f504da61640 1 Processor -- start 2023-12-13T23:55:27.820 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.821+0000 7f504da61640 1 -- start start 2023-12-13T23:55:27.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f504da61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f504815bbe0 0x7f50480cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f504da61640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480cf0f0 con 0x7f504815bfc0 2023-12-13T23:55:27.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f504da61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50480c7ee0 con 0x7f504815f280 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f504da61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50480cf330 con 0x7f504815bbe0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f50467fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f504815bfc0 0x7f50480c5e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43768/0 (socket says 172.21.15.124:43768) 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.822+0000 7f50467fc640 1 -- 172.21.15.124:0/3872322840 learned_addr learned my addr 172.21.15.124:0/3872322840 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5046ffd640 1 --2- 172.21.15.124:0/3872322840 >> v2:172.21.15.203:3300/0 conn(0x7f504815bbe0 0x7f50480cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351388489 0 0) 0x7f503c001110 con 0x7f504815bfc0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5030003520 con 0x7f504815bfc0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 882057377 0 0) 0x7f5030003520 con 0x7f504815bfc0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 >> v2:172.21.15.203:3300/0 conn(0x7f504815bbe0 msgr2=0x7f50480cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 --2- 172.21.15.124:0/3872322840 >> v2:172.21.15.203:3300/0 conn(0x7f504815bbe0 0x7f50480cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 >> v1:172.21.15.203:6789/0 conn(0x7f504815f280 legacy=0x7f504806e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50480c7920 con 0x7f504815bfc0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f503c001800 con 0x7f504815bfc0 2023-12-13T23:55:27.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 >> v1:172.21.15.124:6789/0 conn(0x7f504815bfc0 legacy=0x7f50480c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5030003970 0x7f5030003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50300047c0 0x7f5030004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 0x7f503000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480cf330 con 0x7f5030005180 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503000c120 con 0x7f50300047c0 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503000c2e0 con 0x7f5030003970 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 0x7f503000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.825 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f5046ffd640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5030003970 0x7f5030003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50467fc640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50300047c0 0x7f5030004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 -- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5030003970 msgr2=0x7f5030003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5030003970 0x7f5030003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 -- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50300047c0 msgr2=0x7f5030004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50300047c0 0x7f5030004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.823+0000 7f50477fe640 1 -- 172.21.15.124:0/3872322840 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5030003520 con 0x7f5030005180 2023-12-13T23:55:27.826 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.824+0000 7f50477fe640 1 -- 172.21.15.124:0/3872322840 reap_dead start 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.824+0000 7f50477fe640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 0x7f503000b980 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5040001750 tx=0x7f50300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7e17b2689361d07 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.824+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50400052d0 con 0x7f5030005180 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.824+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504002e050 con 0x7f5030005180 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.824+0000 7f5045ffb640 1 -- 172.21.15.124:0/3872322840 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50400035f0 con 0x7f5030005180 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 -- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 msgr2=0x7f503000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 0x7f503000b980 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5040001750 tx=0x7f50300057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 -- 172.21.15.124:0/3872322840 shutdown_connections 2023-12-13T23:55:27.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 --2- 172.21.15.124:0/3872322840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5030005180 0x7f503000b980 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 -- 172.21.15.124:0/3872322840 >> 172.21.15.124:0/3872322840 conn(0x7f50480b2c10 msgr2=0x7f50480b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:27.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 -- 172.21.15.124:0/3872322840 shutdown_connections 2023-12-13T23:55:27.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.825+0000 7f504da61640 1 -- 172.21.15.124:0/3872322840 wait complete. 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.826+0000 7f504da61640 1 Processor -- start 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.826+0000 7f504da61640 1 -- start start 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 0x7f50480b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50481e5830 0x7f50481e5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481e5230 con 0x7f504815e780 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481e43a0 con 0x7f5030005180 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50481e6200 con 0x7f50481e5830 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57710/0 (socket says 172.21.15.124:57710) 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 -- 172.21.15.124:0/1511348353 learned_addr learned my addr 172.21.15.124:0/1511348353 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:27.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f5046ffd640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 0x7f50480b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50477fe640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50481e5830 0x7f50481e5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 -- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50481e5830 msgr2=0x7f50481e5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50481e5830 0x7f50481e5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 -- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 msgr2=0x7f50480b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 0x7f50480b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 -- 172.21.15.124:0/1511348353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f504815c7e0 con 0x7f504815e780 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f5046ffd640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 0x7f50480b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f50467fc640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f503c007060 tx=0x7f503c007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c007770 con 0x7f504815e780 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503c00f9c0 con 0x7f504815e780 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503c00fd30 con 0x7f504815e780 2023-12-13T23:55:27.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.827+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50481e45b0 con 0x7f504815e780 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.828+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50481e4b70 con 0x7f504815e780 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.829+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5048069400 con 0x7f504815e780 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.829+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f503c01d460 con 0x7f504815e780 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.829+0000 7f502ffff640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 0x7f503406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.829+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f503c089a30 con 0x7f504815e780 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.829+0000 7f5046ffd640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 0x7f503406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.830+0000 7f5046ffd640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 0x7f503406d680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5038007520 tx=0x7f503800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:27.833 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:27.834+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f503c054760 con 0x7f504815e780 2023-12-13T23:55:28.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.038+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50480cf0f0 con 0x7f504815e780 2023-12-13T23:55:28.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.039+0000 7f502ffff640 1 -- 172.21.15.124:0/1511348353 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f50480cf0f0 con 0x7f504815e780 2023-12-13T23:55:28.037 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:28.037 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:28.037 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 msgr2=0x7f503406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 0x7f503406d680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5038007520 tx=0x7f503800b040 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 msgr2=0x7f50480cc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f503c007060 tx=0x7f503c007520 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 shutdown_connections 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f503406b1c0 0x7f503406d680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50481e5830 0x7f50481e5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f504815e780 0x7f50480cc8d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 --2- 172.21.15.124:0/1511348353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5030005180 0x7f50480b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.040+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 >> 172.21.15.124:0/1511348353 conn(0x7f50480b2c10 msgr2=0x7f50480b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:28.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.041+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 shutdown_connections 2023-12-13T23:55:28.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.041+0000 7f504da61640 1 -- 172.21.15.124:0/1511348353 wait complete. 2023-12-13T23:55:28.050 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:28.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.151+0000 7fd2dfe4b640 1 Processor -- start 2023-12-13T23:55:28.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.151+0000 7fd2dfe4b640 1 -- start start 2023-12-13T23:55:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2dfe4b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 0x7fd2d815ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2dfe4b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d8155b70 con 0x7fd2d806a4d0 2023-12-13T23:55:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2dfe4b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d806f3b0 con 0x7fd2d8155790 2023-12-13T23:55:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2dfe4b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d806ee30 con 0x7fd2d815b5e0 2023-12-13T23:55:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2de3e8640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd2d806a4d0 0x7fd2d80714a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:43770/0 (socket says 172.21.15.124:43770) 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2de3e8640 1 -- 172.21.15.124:0/3915836061 learned_addr learned my addr 172.21.15.124:0/3915836061 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.152+0000 7fd2dd3e6640 1 --2- 172.21.15.124:0/3915836061 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 0x7fd2d815ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682596787 0 0) 0x7fd2d8155b70 con 0x7fd2d806a4d0 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4003520 con 0x7fd2d806a4d0 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720956251 0 0) 0x7fd2d806f3b0 con 0x7fd2d8155790 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d8155b70 con 0x7fd2d8155790 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014990989 0 0) 0x7fd2c4003520 con 0x7fd2d806a4d0 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dd3e6640 1 -- 172.21.15.124:0/3915836061 >> v1:172.21.15.203:6789/0 conn(0x7fd2d8155790 legacy=0x7fd2d806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dd3e6640 1 -- 172.21.15.124:0/3915836061 >> v1:172.21.15.124:6789/0 conn(0x7fd2d806a4d0 legacy=0x7fd2d80714a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.153+0000 7fd2dd3e6640 1 -- 172.21.15.124:0/3915836061 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d815c170 con 0x7fd2d815b5e0 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dd3e6640 1 --2- 172.21.15.124:0/3915836061 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 0x7fd2d815ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80097d0 tx=0x7fd2c802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903a48f4843093a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0001670 con 0x7fd2d815b5e0 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c802e050 con 0x7fd2d815b5e0 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dcbe5640 1 -- 172.21.15.124:0/3915836061 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c8030710 con 0x7fd2d815b5e0 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/3915836061 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 msgr2=0x7fd2d815ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/3915836061 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 0x7fd2d815ba60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd2c80097d0 tx=0x7fd2c802b800 comp rx=0 tx=0).stop 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/3915836061 shutdown_connections 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/3915836061 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d815b5e0 0x7fd2d815ba60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/3915836061 >> 172.21.15.124:0/3915836061 conn(0x7fd2d80b1e50 msgr2=0x7fd2d80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/3915836061 shutdown_connections 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.154+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/3915836061 wait complete. 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.155+0000 7fd2dfe4b640 1 Processor -- start 2023-12-13T23:55:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 -- start start 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d806a4d0 0x7fd2d8143be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8155790 0x7fd2d8157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d80ca660 con 0x7fd2d815b5e0 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d8144120 con 0x7fd2d8155790 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dfe4b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d81582d0 con 0x7fd2d806a4d0 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2de3e8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2de3e8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57724/0 (socket says 172.21.15.124:57724) 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2de3e8640 1 -- 172.21.15.124:0/479267341 learned_addr learned my addr 172.21.15.124:0/479267341 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:28.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2dd3e6640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8155790 0x7fd2d8157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.156+0000 7fd2ddbe7640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d806a4d0 0x7fd2d8143be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 -- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d806a4d0 msgr2=0x7fd2d8143be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d806a4d0 0x7fd2d8143be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 -- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8155790 msgr2=0x7fd2d8157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8155790 0x7fd2d8157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 -- 172.21.15.124:0/479267341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c8009400 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2de3e8640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c680 tx=0x7fd2d000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0005ad0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2d0005c30 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d81585b0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0007070 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.157+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d814a110 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.158+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d80ca7a0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.158+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd2d00071d0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.159+0000 7fd2ce7fc640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 0x7fd2bc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.159+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2d0089740 con 0x7fd2d815b5e0 2023-12-13T23:55:28.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.159+0000 7fd2ddbe7640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 0x7fd2bc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.159+0000 7fd2ddbe7640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 0x7fd2bc06d680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8158d00 tx=0x7fd2d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.163+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd2d80ca7a0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.331+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2d80ca7a0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.332+0000 7fd2ce7fc640 1 -- 172.21.15.124:0/479267341 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fd2d80ca7a0 con 0x7fd2d815b5e0 2023-12-13T23:55:28.331 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:28.331 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 msgr2=0x7fd2bc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 0x7fd2bc06d680 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd2d8158d00 tx=0x7fd2d4009040 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 msgr2=0x7fd2d8157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000c680 tx=0x7fd2d000cb40 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 shutdown_connections 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2bc06b1c0 0x7fd2bc06d680 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2d8155790 0x7fd2d8157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2d815b5e0 0x7fd2d8157ca0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 --2- 172.21.15.124:0/479267341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2d806a4d0 0x7fd2d8143be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 >> 172.21.15.124:0/479267341 conn(0x7fd2d80b1e50 msgr2=0x7fd2d80ced00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 shutdown_connections 2023-12-13T23:55:28.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.334+0000 7fd2dfe4b640 1 -- 172.21.15.124:0/479267341 wait complete. 2023-12-13T23:55:28.343 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:28.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.445+0000 7f027833c640 1 Processor -- start 2023-12-13T23:55:28.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.445+0000 7f027833c640 1 -- start start 2023-12-13T23:55:28.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0270144980 0x7f027015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02700ceb70 0x7f02700cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 0x7f02700b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0270071770 con 0x7f02700ceb70 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02700b4ce0 con 0x7f0270144980 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f027833c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02700ccdc0 con 0x7f02700b4370 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02700ceb70 0x7f02700cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02700ceb70 0x7f02700cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57732/0 (socket says 172.21.15.124:57732) 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 -- 172.21.15.124:0/4279739364 learned_addr learned my addr 172.21.15.124:0/4279739364 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02768d9640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 0x7f02700b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02760d8640 1 --2- 172.21.15.124:0/4279739364 >> v2:172.21.15.203:3300/0 conn(0x7f0270144980 0x7f027015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 -- 172.21.15.124:0/4279739364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 msgr2=0x7f02700b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 0x7f02700b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 -- 172.21.15.124:0/4279739364 >> v2:172.21.15.203:3300/0 conn(0x7f0270144980 msgr2=0x7f027015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.446+0000 7f02758d7640 1 --2- 172.21.15.124:0/4279739364 >> v2:172.21.15.203:3300/0 conn(0x7f0270144980 0x7f027015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02758d7640 1 -- 172.21.15.124:0/4279739364 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02700cd0a0 con 0x7f02700ceb70 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02768d9640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 0x7f02700b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02758d7640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700ceb70 0x7f02700cc450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02700b4ec0 tx=0x7f026002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799458640e1e7ddc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02750d6640 1 -- 172.21.15.124:0/4279739364 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260031870 con 0x7f02700ceb70 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02750d6640 1 -- 172.21.15.124:0/4279739364 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02600319d0 con 0x7f02700ceb70 2023-12-13T23:55:28.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.447+0000 7f02750d6640 1 -- 172.21.15.124:0/4279739364 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0260031d60 con 0x7f02700ceb70 2023-12-13T23:55:28.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 -- 172.21.15.124:0/4279739364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700ceb70 msgr2=0x7f02700cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700ceb70 0x7f02700cc450 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f02700b4ec0 tx=0x7f026002b910 comp rx=0 tx=0).stop 2023-12-13T23:55:28.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 -- 172.21.15.124:0/4279739364 shutdown_connections 2023-12-13T23:55:28.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700ceb70 0x7f02700cc450 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 --2- 172.21.15.124:0/4279739364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02700b4370 0x7f02700b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 --2- 172.21.15.124:0/4279739364 >> v2:172.21.15.203:3300/0 conn(0x7f0270144980 0x7f027015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 -- 172.21.15.124:0/4279739364 >> 172.21.15.124:0/4279739364 conn(0x7f027006a4d0 msgr2=0x7f027006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 -- 172.21.15.124:0/4279739364 shutdown_connections 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.448+0000 7f027833c640 1 -- 172.21.15.124:0/4279739364 wait complete. 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.449+0000 7f027833c640 1 Processor -- start 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.449+0000 7f027833c640 1 -- start start 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02700ceb70 0x7f027013e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0270144980 0x7f02701f10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027013d4b0 con 0x7f02700b4370 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027013c600 con 0x7f02700ceb70 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f027833c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027013c8a0 con 0x7f0270144980 2023-12-13T23:55:28.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57734/0 (socket says 172.21.15.124:57734) 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 -- 172.21.15.124:0/267780661 learned_addr learned my addr 172.21.15.124:0/267780661 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02768d9640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0270144980 0x7f02701f10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02758d7640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02700ceb70 0x7f027013e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 -- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0270144980 msgr2=0x7f02701f10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0270144980 0x7f02701f10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 -- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02700ceb70 msgr2=0x7f027013e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02700ceb70 0x7f027013e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 -- 172.21.15.124:0/267780661 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600094d0 con 0x7f02700b4370 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f02760d8640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f026c007e40 tx=0x7f026c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.450+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c014620 con 0x7f02700b4370 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.451+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c00ee90 con 0x7f02700b4370 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.451+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027013cb80 con 0x7f02700b4370 2023-12-13T23:55:28.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.451+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c0149c0 con 0x7f02700b4370 2023-12-13T23:55:28.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.451+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027013d140 con 0x7f02700b4370 2023-12-13T23:55:28.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.452+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027013d2d0 con 0x7f02700b4370 2023-12-13T23:55:28.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.452+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f026c014b20 con 0x7f02700b4370 2023-12-13T23:55:28.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.452+0000 7f0266ffd640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 0x7f025406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:28.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.452+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f026c08a300 con 0x7f02700b4370 2023-12-13T23:55:28.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.453+0000 7f02758d7640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 0x7f025406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:28.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.453+0000 7f02758d7640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 0x7f025406d680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0260036000 tx=0x7f026003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:28.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.457+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f026c055030 con 0x7f02700b4370 2023-12-13T23:55:28.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.657+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0270069400 con 0x7f02700b4370 2023-12-13T23:55:28.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.657+0000 7f0266ffd640 1 -- 172.21.15.124:0/267780661 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f026c058650 con 0x7f02700b4370 2023-12-13T23:55:28.656 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:28.656 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:28.656 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 msgr2=0x7f025406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 0x7f025406d680 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0260036000 tx=0x7f026003c040 comp rx=0 tx=0).stop 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 msgr2=0x7f027013e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f026c007e40 tx=0x7f026c00e4b0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 shutdown_connections 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f025406b1c0 0x7f025406d680 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02700ceb70 0x7f027013e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0270144980 0x7f02701f10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 --2- 172.21.15.124:0/267780661 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02700b4370 0x7f027013e2f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:28.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 >> 172.21.15.124:0/267780661 conn(0x7f027006a4d0 msgr2=0x7f027015c010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:28.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 shutdown_connections 2023-12-13T23:55:28.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:28.659+0000 7f027833c640 1 -- 172.21.15.124:0/267780661 wait complete. 2023-12-13T23:55:29.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-13T23:55:12.125635+0000', 'modified': '2023-12-13T23:55:14.093059+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4308}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4308': {'gid': 4308, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.203:6837/1545879646', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6835', 'nonce': 1545879646}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 1545879646}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701272860131327, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-13T23:55:29.071 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-13T23:55:29.071 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-12-13T23:55:29.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 Processor -- start 2023-12-13T23:55:29.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 -- start start 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69fc0b2b10 con 0x7f69fc15d330 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69fc0d1590 con 0x7f69fc0d2910 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.176+0000 7f6a02c4f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc0d1870 con 0x7f69fc0d3730 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f69fbfff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a009eb640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f69fc15d330 0x7f69fc0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33212/0 (socket says 172.21.15.124:33212) 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a009eb640 1 -- 172.21.15.124:0/382984528 learned_addr learned my addr 172.21.15.124:0/382984528 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f69fbfff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:52060/0 (socket says 172.21.15.124:52060) 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 --2- 172.21.15.124:0/382984528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:60592/0 (socket says 172.21.15.124:60592) 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 -- 172.21.15.124:0/382984528 >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 msgr2=0x7f69fc0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 --2- 172.21.15.124:0/382984528 >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 -- 172.21.15.124:0/382984528 >> v1:172.21.15.203:6789/0 conn(0x7f69fc15d330 legacy=0x7f69fc0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f6a011ec640 1 -- 172.21.15.124:0/382984528 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69fc0d3b90 con 0x7f69fc0d3730 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f69fb7fe640 1 -- 172.21.15.124:0/382984528 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139208122 0 0) 0x7f69fc0b2b10 con 0x7f69fc15d330 2023-12-13T23:55:29.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.177+0000 7f69fbfff640 1 --2- 172.21.15.124:0/382984528 >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:29.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f6a011ec640 1 --2- 172.21.15.124:0/382984528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a416f7252f1e1533 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f69fb7fe640 1 -- 172.21.15.124:0/382984528 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f402cbc0 con 0x7f69fc0d3730 2023-12-13T23:55:29.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f69fb7fe640 1 -- 172.21.15.124:0/382984528 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f402cd20 con 0x7f69fc0d3730 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f69fb7fe640 1 -- 172.21.15.124:0/382984528 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4034660 con 0x7f69fc0d3730 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f6a02c4f640 1 -- 172.21.15.124:0/382984528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 msgr2=0x7f69fc0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.178+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/382984528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f69f4009d90 tx=0x7f69f402be10 comp rx=0 tx=0).stop 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 -- 172.21.15.124:0/382984528 shutdown_connections 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/382984528 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc0d3730 0x7f69fc0b6390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/382984528 >> v2:172.21.15.203:3300/0 conn(0x7f69fc0d2910 0x7f69fc0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 -- 172.21.15.124:0/382984528 >> 172.21.15.124:0/382984528 conn(0x7f69fc072020 msgr2=0x7f69fc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 -- 172.21.15.124:0/382984528 shutdown_connections 2023-12-13T23:55:29.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.179+0000 7f6a02c4f640 1 -- 172.21.15.124:0/382984528 wait complete. 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.180+0000 7f6a02c4f640 1 Processor -- start 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.180+0000 7f6a02c4f640 1 -- start start 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.180+0000 7f6a02c4f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69fc0d2910 0x7f69fc15e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.180+0000 7f6a02c4f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.180+0000 7f6a02c4f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc15d330 0x7f69fc1f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a02c4f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc15eca0 con 0x7f69fc0d3730 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a02c4f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc15ef40 con 0x7f69fc0d2910 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a02c4f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69fc15f220 con 0x7f69fc15d330 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57736/0 (socket says 172.21.15.124:57736) 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 -- 172.21.15.124:0/2403247892 learned_addr learned my addr 172.21.15.124:0/2403247892 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a009eb640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69fc0d2910 0x7f69fc15e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a011ec640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc15d330 0x7f69fc1f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 -- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc15d330 msgr2=0x7f69fc1f3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc15d330 0x7f69fc1f3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 -- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69fc0d2910 msgr2=0x7f69fc15e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69fc0d2910 0x7f69fc15e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 -- 172.21.15.124:0/2403247892 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e8009400 con 0x7f69fc0d3730 2023-12-13T23:55:29.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69fbfff640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e960 tx=0x7f69e800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69f97fa640 1 -- 172.21.15.124:0/2403247892 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e800ca00 con 0x7f69fc0d3730 2023-12-13T23:55:29.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69f97fa640 1 -- 172.21.15.124:0/2403247892 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e800cb60 con 0x7f69fc0d3730 2023-12-13T23:55:29.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f4009a70 con 0x7f69fc0d3730 2023-12-13T23:55:29.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f69f97fa640 1 -- 172.21.15.124:0/2403247892 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8013470 con 0x7f69fc0d3730 2023-12-13T23:55:29.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.181+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc15f7e0 con 0x7f69fc0d3730 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.183+0000 7f69f97fa640 1 -- 172.21.15.124:0/2403247892 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f69e80135d0 con 0x7f69fc0d3730 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.183+0000 7f69f97fa640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 0x7f69e006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.183+0000 7f69f97fa640 1 -- 172.21.15.124:0/2403247892 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e8089c40 con 0x7f69fc0d3730 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.183+0000 7f6a009eb640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 0x7f69e006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.184+0000 7f6a009eb640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 0x7f69e006d680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69f0000f60 tx=0x7f69f0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.184+0000 7f6a057e6740 1 Processor -- start 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.184+0000 7f6a057e6740 1 -- start start 2023-12-13T23:55:29.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea44ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea45000 0x55a0bea74b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0be9b8020 con 0x55a0bea71c70 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0be9b5710 con 0x55a0bea45000 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0be9b1000 con 0x55a0bea71890 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a009eb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea44ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57750/0 (socket says 172.21.15.124:57750) 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 -- 172.21.15.124:0/1931628544 learned_addr learned my addr 172.21.15.124:0/1931628544 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f6a011ec640 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea45000 0x55a0bea74b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 -- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 msgr2=0x55a0bea44ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea44ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 -- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea45000 msgr2=0x55a0bea74b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea45000 0x55a0bea74b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 -- 172.21.15.124:0/1931628544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0be973c90 con 0x55a0bea71c70 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69e8060400 tx=0x7f69e8058040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79b01299466bb1f0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69e67fc640 1 -- 172.21.15.124:0/1931628544 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8060930 con 0x55a0bea71c70 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69e67fc640 1 -- 172.21.15.124:0/1931628544 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8056a80 con 0x55a0bea71c70 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.185+0000 7f69e67fc640 1 -- 172.21.15.124:0/1931628544 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e80604a0 con 0x55a0bea71c70 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- 172.21.15.124:0/1931628544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 msgr2=0x55a0bea74620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f69e8060400 tx=0x7f69e8058040 comp rx=0 tx=0).stop 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- 172.21.15.124:0/1931628544 shutdown_connections 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea45000 0x55a0bea74b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea74620 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1931628544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea44ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- 172.21.15.124:0/1931628544 >> 172.21.15.124:0/1931628544 conn(0x55a0bea46d30 msgr2=0x55a0bea8f620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- 172.21.15.124:0/1931628544 shutdown_connections 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- 172.21.15.124:0/1931628544 wait complete. 2023-12-13T23:55:29.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.186+0000 7f6a057e6740 1 -- start start 2023-12-13T23:55:29.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 Processor -- start 2023-12-13T23:55:29.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea613d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 0x55a0bea86cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0be96ae00 con 0x55a0bea71c70 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0be96aaa0 con 0x55a0bea75610 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0be930270 con 0x55a0bea71890 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f6a057e6740 10 client.0 _ll_register_callbacks cb 0x55a0be6c2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57754/0 (socket says 172.21.15.124:57754) 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.187+0000 7f69fbfff640 1 -- 172.21.15.124:0/1657360033 learned_addr learned my addr 172.21.15.124:0/1657360033 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea613d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a011ec640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 0x55a0bea86cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 -- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 msgr2=0x55a0bea613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea613d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 -- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 msgr2=0x55a0bea86cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 0x55a0bea86cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 -- 172.21.15.124:0/1657360033 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69fc15f7e0 con 0x55a0bea71c70 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a011ec640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 0x55a0bea86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f69e801f040 tx=0x7f69e8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb46a2a547de1a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8060930 con 0x55a0bea71c70 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e80544d0 con 0x55a0bea71c70 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0be9a9e90 con 0x55a0bea71c70 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69e8054690 con 0x55a0bea71c70 2023-12-13T23:55:29.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 10 client.4698 fetch_fsmap learned FSMap version 6 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55a0be9af680 con 0x55a0bea71c70 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a0be9a9e90 con 0x55a0bea71c70 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69e8011070 con 0x55a0bea71c70 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 10 client.4698 fetch_fsmap finished waiting for FSMap version 6 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 10 client.4698 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 20 client.4698 populate_metadata read hostname 'smithi124' 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0bea91eb0 0x55a0bea775c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 4 client.4698 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.188+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 --> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0be9a9bb0 con 0x55a0bea91eb0 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.189+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0bea91eb0 0x55a0bea775c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.189+0000 7f69db7fe640 10 client.4698 ms_handle_connect on v2:172.21.15.203:6835/1545879646 2023-12-13T23:55:29.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.189+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0bea91eb0 0x55a0bea775c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a1ab029d40182c server_cookie=4050823bc9eb6044 in_seq=0 out_seq=0 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f69db7fe640 1 -- 172.21.15.124:0/1657360033 <== mds.0 v2:172.21.15.203:6835/1545879646 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0be9a9bb0 con 0x55a0bea91eb0 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f69db7fe640 10 client.4698 handle_command_reply: tid=0 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 client.4698 shutdown 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 msgr2=0x55a0bea60b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f69e801f040 tx=0x7f69e8000bc0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 shutdown_connections 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0bea91eb0 0x55a0bea775c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea75610 0x55a0bea86cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea60b30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f69db7fe640 0 client.4698 ms_handle_reset on v2:172.21.15.203:6835/1545879646 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1657360033 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea71890 0x55a0bea613d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 >> 172.21.15.124:0/1657360033 conn(0x55a0bea45000 msgr2=0x55a0bea48850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 shutdown_connections 2023-12-13T23:55:29.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 1 -- 172.21.15.124:0/1657360033 wait complete. 2023-12-13T23:55:29.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.190+0000 7f6a057e6740 20 client.4698 trim_cache size 0 max 16384 2023-12-13T23:55:29.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.205+0000 7f6a057e6740 1 Processor -- start 2023-12-13T23:55:29.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.205+0000 7f6a057e6740 1 -- start start 2023-12-13T23:55:29.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71c70 0x55a0beaa8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0beaa8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0bea44020 con 0x55a0bea71890 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0bea601d0 con 0x55a0bea71c70 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0bea44780 con 0x55a0bea75610 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57762/0 (socket says 172.21.15.124:57762) 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 -- 172.21.15.124:0/465843453 learned_addr learned my addr 172.21.15.124:0/465843453 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 -- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 msgr2=0x55a0beaa8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0beaa8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 -- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71c70 msgr2=0x55a0beaa8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71c70 0x55a0beaa8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.206+0000 7f6a009eb640 1 -- 172.21.15.124:0/465843453 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0be94cd00 con 0x55a0bea71890 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a009eb640 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0b4a90 tx=0x7f69f0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42ff047bdd364e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f69e6ffd640 1 -- 172.21.15.124:0/465843453 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f00287a0 con 0x55a0bea71890 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f69e6ffd640 1 -- 172.21.15.124:0/465843453 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f0028900 con 0x55a0bea71890 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f69e6ffd640 1 -- 172.21.15.124:0/465843453 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0028b00 con 0x55a0bea71890 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 -- 172.21.15.124:0/465843453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 msgr2=0x55a0bea8fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f69fc0b4a90 tx=0x7f69f0013000 comp rx=0 tx=0).stop 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 -- 172.21.15.124:0/465843453 shutdown_connections 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71c70 0x55a0beaa8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0beaa8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 --2- 172.21.15.124:0/465843453 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71890 0x55a0bea8fe60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 -- 172.21.15.124:0/465843453 >> 172.21.15.124:0/465843453 conn(0x55a0bea45000 msgr2=0x55a0bea86a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 -- 172.21.15.124:0/465843453 shutdown_connections 2023-12-13T23:55:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.207+0000 7f6a057e6740 1 -- 172.21.15.124:0/465843453 wait complete. 2023-12-13T23:55:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.208+0000 7f6a057e6740 1 -- start start 2023-12-13T23:55:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.208+0000 7f6a057e6740 1 Processor -- start 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71890 0x55a0bea9e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0bea481e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0bea492e0 con 0x55a0bea71c70 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0beae25e0 con 0x55a0bea71890 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0bea490d0 con 0x55a0bea75610 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 10 client.0 _ll_register_callbacks cb 0x7f69f4037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:55:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57768/0 (socket says 172.21.15.124:57768) 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 -- 172.21.15.124:0/1804871105 learned_addr learned my addr 172.21.15.124:0/1804871105 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71890 0x55a0bea9e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a011ec640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0bea481e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 -- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 msgr2=0x55a0bea481e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0bea481e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 -- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71890 msgr2=0x55a0bea9e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71890 0x55a0bea9e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 -- 172.21.15.124:0/1804871105 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f0004d90 con 0x55a0bea71c70 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a011ec640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0bea481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69fbfff640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f69e8054be0 tx=0x7f69e8088040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e697a34b40685358 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e8060930 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e80880b0 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.209+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69e8056a80 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69e800e070 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 10 client.4704 fetch_fsmap learned FSMap version 6 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x55a0be9724f0 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e8056a80 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 6) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f69e8011040 con 0x55a0bea71c70 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 10 client.4704 fetch_fsmap finished waiting for FSMap version 6 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 10 client.4704 resolve_mds: resolved name 'd' to daemon mds.d 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 20 client.4704 populate_metadata read hostname 'smithi124' 2023-12-13T23:55:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0beae2b20 0x55a0bea48720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 4 client.4704 mds_command: new command op to 4308 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 --> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a0be6b16c0 con 0x55a0beae2b20 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0beae2b20 0x55a0bea48720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f69db7fe640 10 client.4704 ms_handle_connect on v2:172.21.15.203:6835/1545879646 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.210+0000 7f6a009eb640 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0beae2b20 0x55a0bea48720 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d4dee43f996ba6 server_cookie=364ee3c2115ad715 in_seq=0 out_seq=0 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f69db7fe640 1 -- 172.21.15.124:0/1804871105 <== mds.0 v2:172.21.15.203:6835/1545879646 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55a0be6b16c0 con 0x55a0beae2b20 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f69db7fe640 10 client.4704 handle_command_reply: tid=0 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 client.4704 shutdown 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 msgr2=0x55a0bea6de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f69e8054be0 tx=0x7f69e8088040 comp rx=0 tx=0).stop 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 shutdown_connections 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:6835/1545879646,v1:172.21.15.203:6837/1545879646] conn(0x55a0beae2b20 0x55a0bea48720 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x55a0bea75610 0x55a0bea481e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f69db7fe640 0 client.4704 ms_handle_reset on v2:172.21.15.203:6835/1545879646 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a0bea71c70 0x55a0bea6de50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 --2- 172.21.15.124:0/1804871105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x55a0bea71890 0x55a0bea9e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 >> 172.21.15.124:0/1804871105 conn(0x55a0bea45000 msgr2=0x55a0bea6f390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 shutdown_connections 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 1 -- 172.21.15.124:0/1804871105 wait complete. 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a057e6740 20 client.4704 trim_cache size 0 max 16384 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:55:29.217 INFO:teuthology.orchestra.run.smithi124.stdout: "cluster_fsid": "f45f9ea9-91cf-4589-b298-b4d7e1fc4f09", 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "whoami": 0, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 4308, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "want_state": "up:active", 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "state": "up:active", 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs", 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "rank_uptime": 16.461665107999998, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "mdsmap_epoch": 6, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch": 21, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch_barrier": 21, 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout: "uptime": 23.444968788000001 2023-12-13T23:55:29.218 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 msgr2=0x7f69e006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 0x7f69e006d680 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f69f0000f60 tx=0x7f69f0007d50 comp rx=0 tx=0).stop 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 msgr2=0x7f69fc1f3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f69e800e960 tx=0x7f69e800ee20 comp rx=0 tx=0).stop 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 shutdown_connections 2023-12-13T23:55:29.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e006b1c0 0x7f69e006d680 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f69fc15d330 0x7f69fc1f3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69fc0d3730 0x7f69fc1f3520 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 --2- 172.21.15.124:0/2403247892 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69fc0d2910 0x7f69fc15e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 >> 172.21.15.124:0/2403247892 conn(0x7f69fc072020 msgr2=0x7f69fc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 shutdown_connections 2023-12-13T23:55:29.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.211+0000 7f6a02c4f640 1 -- 172.21.15.124:0/2403247892 wait complete. 2023-12-13T23:55:29.221 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-12-13T23:55:29.245 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-12-13T23:55:29.245 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-12-13T23:55:29.246 INFO:tasks.ceph_fuse:client.0 config is {} 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:29.246 INFO:tasks.ceph_fuse:client.2 config is {} 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:29.246 INFO:tasks.ceph_fuse:client.1 config is {} 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-13T23:55:29.246 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:29.246 INFO:tasks.ceph_fuse:client.3 config is {} 2023-12-13T23:55:29.247 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-12-13T23:55:29.247 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:29.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.247 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:55:29.262 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.263 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-12-13T23:55:29.331 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2023-12-13T23:55:29.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:55:29.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.332 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2023-12-13T23:55:29.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link delete ceph-brx 2023-12-13T23:55:29.424 INFO:teuthology.orchestra.run.smithi203.stderr:Cannot find device "ceph-brx" 2023-12-13T23:55:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:55:29.426 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-12-13T23:55:29.426 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:55:29.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.526+0000 7f7dd08e7640 1 Processor -- start 2023-12-13T23:55:29.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.526+0000 7f7dd08e7640 1 -- start start 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dd08e7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dd08e7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc06bbf0 0x7f7dcc06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dd08e7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dcc0adae0 con 0x7f7dcc15d1d0 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dd08e7640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dcc0addc0 con 0x7f7dcc0ad640 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dd08e7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc0ae0a0 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.527+0000 7f7dc9d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57778/0 (socket says 172.21.15.124:57778) 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 -- 172.21.15.124:0/2816924485 learned_addr learned my addr 172.21.15.124:0/2816924485 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dcad9d640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc06bbf0 0x7f7dcc06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 -- 172.21.15.124:0/2816924485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc06bbf0 msgr2=0x7f7dcc06d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc06bbf0 0x7f7dcc06d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 -- 172.21.15.124:0/2816924485 >> v1:172.21.15.203:6789/0 conn(0x7f7dcc15d1d0 legacy=0x7f7dcc06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 -- 172.21.15.124:0/2816924485 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0ae380 con 0x7f7dcc0ad640 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.528+0000 7f7dc9d9b640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009e50 tx=0x7f7dc002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fa0f392814f2c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dc959a640 1 -- 172.21.15.124:0/2816924485 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc002cb00 con 0x7f7dcc0ad640 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dc959a640 1 -- 172.21.15.124:0/2816924485 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc002cc60 con 0x7f7dcc0ad640 2023-12-13T23:55:29.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dc959a640 1 -- 172.21.15.124:0/2816924485 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc00343f0 con 0x7f7dcc0ad640 2023-12-13T23:55:29.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 -- 172.21.15.124:0/2816924485 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc0ad640 msgr2=0x7f7dcc06d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0009e50 tx=0x7f7dc002bf60 comp rx=0 tx=0).stop 2023-12-13T23:55:29.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 -- 172.21.15.124:0/2816924485 shutdown_connections 2023-12-13T23:55:29.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc06bbf0 0x7f7dcc06d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/2816924485 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc0ad640 0x7f7dcc06d0b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 -- 172.21.15.124:0/2816924485 >> 172.21.15.124:0/2816924485 conn(0x7f7dcc070e70 msgr2=0x7f7dcc071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 -- 172.21.15.124:0/2816924485 shutdown_connections 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.529+0000 7f7dd08e7640 1 -- 172.21.15.124:0/2816924485 wait complete. 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.530+0000 7f7dd08e7640 1 Processor -- start 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 -- start start 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc0ad640 0x7f7dcc157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dcc15d1d0 0x7f7dcc149770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc158240 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc15ef80 con 0x7f7dcc15d1d0 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dd08e7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dcc15f260 con 0x7f7dcc0ad640 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dca59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dca59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57780/0 (socket says 172.21.15.124:57780) 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dca59c640 1 -- 172.21.15.124:0/3861746375 learned_addr learned my addr 172.21.15.124:0/3861746375 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dc9d9b640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc0ad640 0x7f7dcc157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.531+0000 7f7dcad9d640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dcc15d1d0 0x7f7dcc149770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 -- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc0ad640 msgr2=0x7f7dcc157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc0ad640 0x7f7dcc157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 -- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dcc15d1d0 msgr2=0x7f7dcc149770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dcc15d1d0 0x7f7dcc149770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 -- 172.21.15.124:0/3861746375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc0009b30 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dca59c640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c400 tx=0x7f7dc400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40052d0 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc4005430 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dcc15f540 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4003830 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.532+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dcc15fb00 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.533+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dcc15fcf0 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.533+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7dc4003990 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.534+0000 7f7dbaffd640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 0x7f7dac06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:29.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.534+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7dc4088fa0 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.534+0000 7f7dc9d9b640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 0x7f7dac06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:29.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.534+0000 7f7dc9d9b640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 0x7f7dac06d680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0035000 tx=0x7f7dc0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:29.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.538+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7dc40bbe90 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.739+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7dcc069400 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.739+0000 7f7dbaffd640 1 -- 172.21.15.124:0/3861746375 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7dc4017070 con 0x7f7dcc06bbf0 2023-12-13T23:55:29.738 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 msgr2=0x7f7dac06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 0x7f7dac06d680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0035000 tx=0x7f7dc0030000 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 msgr2=0x7f7dcc15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c400 tx=0x7f7dc400c8c0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 shutdown_connections 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dac06b1c0 0x7f7dac06d680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7dcc15d1d0 0x7f7dcc149770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7dcc0ad640 0x7f7dcc157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 --2- 172.21.15.124:0/3861746375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7dcc06bbf0 0x7f7dcc15e930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 >> 172.21.15.124:0/3861746375 conn(0x7f7dcc070e70 msgr2=0x7f7dcc06d140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:29.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 shutdown_connections 2023-12-13T23:55:29.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:29.741+0000 7f7dd08e7640 1 -- 172.21.15.124:0/3861746375 wait complete. 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:55:29.750 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:55:29.751 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:55:29.751 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:55:29.751 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.751 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:55:29.765 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:29.765 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:55:29.766 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:29.767 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:55:29.767 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42677sec preferred_lft 42677sec 2023-12-13T23:55:29.767 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:55:29.767 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:29.767 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-13T23:55:29.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.767 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add name ceph-brx type bridge 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr flush dev ceph-brx 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set ceph-brx up 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-13T23:55:29.768 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:29.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.923 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:55:29.950 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:55:29.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:29.951 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:55:30.013 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:55:30.013 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:55:30.013 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:55:30.013 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:55:30.014 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:55:30.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.014 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:30.014 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:30.014 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:55:30.014 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:55:30.014 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:55:30.015 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:30.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.323 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:55:30.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.339 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2023-12-13T23:55:30.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.394 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:30.394 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:30.394 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:30.394 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-12-13T23:55:30.394 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:30.477 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-12-13T23:55:30.477 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.477 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:30.477 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-12-13T23:55:30.481 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:30.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.615 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:30.615 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:30.615 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:55:30.615 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.0 master ceph-brx 2023-12-13T23:55:30.615 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:30.656 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:55:30.657 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:30.657 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:30.710 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:55:30.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:30.710 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:30.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:55:30.833 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:55:30.889 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:30.890 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:30.891 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:30.891 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:55:30.944 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:55:30.944 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:55:30.986 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:55:31.014 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:31.070 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:55:31.070 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:55:31.070 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:55:31.070 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:55:31.071 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:55:31.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:31.073 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:55:31.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:31.075 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:31.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.075 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:55:31.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:55:31.081+0000 7f0a5b243480 -1 init, newargv = 0x55be1df4e750 newargc=15 2023-12-13T23:55:31.080 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[97846]: starting ceph client 2023-12-13T23:55:31.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[97846]: starting fuse 2023-12-13T23:55:31.096 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:55:31.096 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:55:31.096 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:55:31.096 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:55:31.097 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:55:31.098 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:31.099 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:55:31.099 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:55:31.216 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.97846.asok 2023-12-13T23:55:31.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:55:31.227 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.97846.asok 2023-12-13T23:55:31.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.227 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.97846.asok status 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:55:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "97846", 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 4713, 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 4713 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:55:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3829749990 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3829749990 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.4713 192.168.0.1:0/3829749990", 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/3829749990", 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 6, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 21, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:55:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:55:31.369 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:55:31.369 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:55:31.374 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:55:31.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.475+0000 7f340534c640 1 Processor -- start 2023-12-13T23:55:31.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.475+0000 7f340534c640 1 -- start start 2023-12-13T23:55:31.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f340534c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3400144980 0x7f340015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f340534c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 0x7f34000cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f340534c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3400071770 con 0x7f34000b4370 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f340534c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34000ccdc0 con 0x7f3400144980 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f340534c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34000cd0a0 con 0x7f34000ceb70 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3400144980 0x7f340015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3400144980 0x7f340015c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51260/0 (socket says 172.21.15.124:51260) 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 -- 172.21.15.124:0/624793621 learned_addr learned my addr 172.21.15.124:0/624793621 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33fe7fc640 1 --2- 172.21.15.124:0/624793621 >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 0x7f34000cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 -- 172.21.15.124:0/624793621 >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 msgr2=0x7f34000cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 --2- 172.21.15.124:0/624793621 >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 0x7f34000cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 -- 172.21.15.124:0/624793621 >> v1:172.21.15.203:6789/0 conn(0x7f34000b4370 legacy=0x7f3400071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33feffd640 1 -- 172.21.15.124:0/624793621 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34000cd380 con 0x7f3400144980 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.476+0000 7f33fe7fc640 1 --2- 172.21.15.124:0/624793621 >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 0x7f34000cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.477+0000 7f33feffd640 1 --2- 172.21.15.124:0/624793621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f340015c7c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f340015b600 tx=0x7f33f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5623b19a2a34c51e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:31.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.477+0000 7f33fdffb640 1 -- 172.21.15.124:0/624793621 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f002ca80 con 0x7f3400144980 2023-12-13T23:55:31.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.477+0000 7f33fdffb640 1 -- 172.21.15.124:0/624793621 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f002cbe0 con 0x7f3400144980 2023-12-13T23:55:31.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.477+0000 7f33fdffb640 1 -- 172.21.15.124:0/624793621 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f00343f0 con 0x7f3400144980 2023-12-13T23:55:31.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 -- 172.21.15.124:0/624793621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 msgr2=0x7f340015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 --2- 172.21.15.124:0/624793621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f340015c7c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f340015b600 tx=0x7f33f002bee0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 -- 172.21.15.124:0/624793621 shutdown_connections 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 --2- 172.21.15.124:0/624793621 >> v2:172.21.15.203:3300/0 conn(0x7f34000ceb70 0x7f34000cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 --2- 172.21.15.124:0/624793621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f340015c7c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 -- 172.21.15.124:0/624793621 >> 172.21.15.124:0/624793621 conn(0x7f340006a4d0 msgr2=0x7f340006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 -- 172.21.15.124:0/624793621 shutdown_connections 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.478+0000 7f340534c640 1 -- 172.21.15.124:0/624793621 wait complete. 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.479+0000 7f340534c640 1 Processor -- start 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.479+0000 7f340534c640 1 -- start start 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34000b4370 0x7f3400142af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f34000ceb70 0x7f34000a9350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3400156a20 con 0x7f3400144980 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34001566d0 con 0x7f34000b4370 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f340534c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34000a8570 con 0x7f34000ceb70 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51268/0 (socket says 172.21.15.124:51268) 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 -- 172.21.15.124:0/3270789119 learned_addr learned my addr 172.21.15.124:0/3270789119 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33fe7fc640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f34000ceb70 0x7f34000a9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33feffd640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34000b4370 0x7f3400142af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 -- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f34000ceb70 msgr2=0x7f34000a9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f34000ceb70 0x7f34000a9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 -- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34000b4370 msgr2=0x7f3400142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34000b4370 0x7f3400142af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 -- 172.21.15.124:0/3270789119 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f0009b30 con 0x7f3400144980 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.480+0000 7f33ff7fe640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f33f800c410 tx=0x7f33f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.481+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8011790 con 0x7f3400144980 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.481+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f80118f0 con 0x7f3400144980 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.481+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34000a8850 con 0x7f3400144980 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.481+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f8011ce0 con 0x7f3400144980 2023-12-13T23:55:31.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.481+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34000b16b0 con 0x7f3400144980 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.482+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3400156b60 con 0x7f3400144980 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.482+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f33f801c460 con 0x7f3400144980 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.482+0000 7f33f77fe640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 0x7f33e006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.482+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33f8089850 con 0x7f3400144980 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.483+0000 7f33feffd640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 0x7f33e006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:31.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.483+0000 7f33feffd640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 0x7f33e006d680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f33f0035000 tx=0x7f33f0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:31.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.487+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3400156b60 con 0x7f3400144980 2023-12-13T23:55:31.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.691+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3400156b60 con 0x7f3400144980 2023-12-13T23:55:31.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.691+0000 7f33f77fe640 1 -- 172.21.15.124:0/3270789119 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3400156b60 con 0x7f3400144980 2023-12-13T23:55:31.690 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.693+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 msgr2=0x7f33e006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.693+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 0x7f33e006d680 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f33f0035000 tx=0x7f33f0030000 comp rx=0 tx=0).stop 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.693+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 msgr2=0x7f34000a9890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.693+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f33f800c410 tx=0x7f33f800c8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 shutdown_connections 2023-12-13T23:55:31.692 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f33e006b1c0 0x7f33e006d680 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3400144980 0x7f34000a9890 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f34000ceb70 0x7f34000a9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 --2- 172.21.15.124:0/3270789119 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f34000b4370 0x7f3400142af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 >> 172.21.15.124:0/3270789119 conn(0x7f340006a4d0 msgr2=0x7f3400071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 shutdown_connections 2023-12-13T23:55:31.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:31.694+0000 7f340534c640 1 -- 172.21.15.124:0/3270789119 wait complete. 2023-12-13T23:55:31.702 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:55:31.703 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-12-13T23:55:31.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.703 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:55:31.718 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:31.719 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42675sec preferred_lft 42675sec 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:55:31.720 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:31.721 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link tentative 2023-12-13T23:55:31.721 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:31.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.721 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:55:31.784 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:55:31.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.786 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:55:31.842 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:55:31.842 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:55:31.842 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:55:31.843 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:55:31.843 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:55:31.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.843 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:31.843 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:31.843 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:55:31.843 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:55:31.844 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:55:31.844 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:31.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.938 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:55:31.953 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:55:31.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:31.954 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2023-12-13T23:55:32.008 INFO:teuthology.orchestra.run.smithi124.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-12-13T23:55:32.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.009 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:32.009 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:32.009 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:55:32.009 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-12-13T23:55:32.009 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:32.090 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.090 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 02:c5:c5:c4:8c:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-13T23:55:32.157 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.158 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::c5:c5ff:fec4:8cd7/64 scope link 2023-12-13T23:55:32.158 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.159 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:32.226 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 02:c5:c5:c4:8c:d7 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-13T23:55:32.227 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-13T23:55:32.227 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.227 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::c5:c5ff:fec4:8cd7/64 scope link 2023-12-13T23:55:32.227 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:32.228 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-12-13T23:55:32.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-12-13T23:55:32.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-12-13T23:55:32.229 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-12-13T23:55:32.229 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:32.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.352 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:55:32.352 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:55:32.352 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.1 up 2023-12-13T23:55:32.352 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.1 master ceph-brx 2023-12-13T23:55:32.352 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:55:32.394 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2023-12-13T23:55:32.394 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:32.394 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:32.449 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-12-13T23:55:32.449 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:32.449 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:32.503 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:55:32.570 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:32.625 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:55:32.625 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:55:32.626 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:32.627 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:32.628 INFO:teuthology.orchestra.run.smithi124.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:32.628 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:55:32.628 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:55:32.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.628 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:55:32.681 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:55:32.681 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2023-12-13T23:55:32.681 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2023-12-13T23:55:32.723 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:55:32.751 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:32.807 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:32.808 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:55:32.809 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:55:32.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.810 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:55:32.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi124.stderr:2023-12-13T23:55:32.810+0000 7f9746c3e480 -1 init, newargv = 0x55be03723a60 newargc=15 2023-12-13T23:55:32.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi124.stderr:ceph-fuse[98339]: starting ceph client 2023-12-13T23:55:32.813 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi124.stderr:ceph-fuse[98339]: starting fuse 2023-12-13T23:55:32.825 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:55:32.826 INFO:teuthology.orchestra.run.smithi124.stdout:44 2023-12-13T23:55:32.826 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:55:32.826 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:55:32.827 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:55:32.828 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:55:32.828 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.2")) 2023-12-13T23:55:32.828 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:55:32.945 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.2.98339.asok 2023-12-13T23:55:32.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:55:32.955 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.98339.asok 2023-12-13T23:55:32.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:32.955 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.98339.asok status 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "2", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "98339", 2023-12-13T23:55:33.089 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 4725, 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 4725 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:55:33.090 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.2:0", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1957326655 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.2:0", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1957326655 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.4725 192.168.0.2:0/1957326655", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.2:0/1957326655", 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 6, 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 21, 2023-12-13T23:55:33.091 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:55:33.092 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:55:33.092 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:55:33.092 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:55:33.097 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:55:33.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.197+0000 7f2313e75640 1 Processor -- start 2023-12-13T23:55:33.196 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 -- start start 2023-12-13T23:55:33.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 0x7f230c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f230c15dec0 0x7f230c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f230c0b49f0 con 0x7f230c150560 2023-12-13T23:55:33.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f230c0b3560 con 0x7f230c15dec0 2023-12-13T23:55:33.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.198+0000 7f2313e75640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c0b3de0 con 0x7f230c150940 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f230c15dec0 0x7f230c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f230c15dec0 0x7f230c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51272/0 (socket says 172.21.15.124:51272) 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2311c11640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f230c150560 0x7f230c0b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39710/0 (socket says 172.21.15.124:39710) 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 -- 172.21.15.124:0/896626863 learned_addr learned my addr 172.21.15.124:0/896626863 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2311410640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 0x7f230c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 -- 172.21.15.124:0/896626863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 msgr2=0x7f230c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 0x7f230c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 -- 172.21.15.124:0/896626863 >> v1:172.21.15.124:6789/0 conn(0x7f230c150560 legacy=0x7f230c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2312412640 1 -- 172.21.15.124:0/896626863 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f230c0b40c0 con 0x7f230c15dec0 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2310c0f640 1 -- 172.21.15.124:0/896626863 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238395042 0 0) 0x7f230c0b49f0 con 0x7f230c150560 2023-12-13T23:55:33.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.199+0000 7f2311410640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 0x7f230c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:33.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2312412640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c15dec0 0x7f230c15d740 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2308009870 tx=0x7f230802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10db11ccf96f5f33 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:33.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2310c0f640 1 -- 172.21.15.124:0/896626863 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308030840 con 0x7f230c15dec0 2023-12-13T23:55:33.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2310c0f640 1 -- 172.21.15.124:0/896626863 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23080309a0 con 0x7f230c15dec0 2023-12-13T23:55:33.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2310c0f640 1 -- 172.21.15.124:0/896626863 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308030d30 con 0x7f230c15dec0 2023-12-13T23:55:33.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 -- 172.21.15.124:0/896626863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c15dec0 msgr2=0x7f230c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c15dec0 0x7f230c15d740 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f2308009870 tx=0x7f230802b980 comp rx=0 tx=0).stop 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 -- 172.21.15.124:0/896626863 shutdown_connections 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c150940 0x7f230c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 --2- 172.21.15.124:0/896626863 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c15dec0 0x7f230c15d740 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 -- 172.21.15.124:0/896626863 >> 172.21.15.124:0/896626863 conn(0x7f230c06a4d0 msgr2=0x7f230c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 -- 172.21.15.124:0/896626863 shutdown_connections 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.200+0000 7f2313e75640 1 -- 172.21.15.124:0/896626863 wait complete. 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 Processor -- start 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 -- start start 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 0x7f230c1f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 0x7f230c14b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c1ef700 con 0x7f230c14bb80 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230c1eea80 con 0x7f230c150560 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2313e75640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f230c1eed60 con 0x7f230c15dec0 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2312412640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2312412640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51280/0 (socket says 172.21.15.124:51280) 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.202+0000 7f2312412640 1 -- 172.21.15.124:0/4043642408 learned_addr learned my addr 172.21.15.124:0/4043642408 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2311410640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 0x7f230c14b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2311c11640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 0x7f230c1f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 -- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 msgr2=0x7f230c14b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 0x7f230c14b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 -- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 msgr2=0x7f230c1f1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 0x7f230c1f1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 -- 172.21.15.124:0/4043642408 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080094d0 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2311410640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 0x7f230c14b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2311c11640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 0x7f230c1f1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2312412640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2308009870 tx=0x7f2308004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308039a90 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2308039c10 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f230c1ef040 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2308043410 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.203+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230c1ee740 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.204+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c1ef840 con 0x7f230c14bb80 2023-12-13T23:55:33.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.205+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f230804c3b0 con 0x7f230c14bb80 2023-12-13T23:55:33.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.205+0000 7f23027fc640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 0x7f22f006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:33.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.205+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23080abee0 con 0x7f230c14bb80 2023-12-13T23:55:33.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.205+0000 7f2311c11640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 0x7f22f006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:33.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.206+0000 7f2311c11640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 0x7f22f006d680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23040012e0 tx=0x7f2304007f90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:33.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.209+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f230c1ef840 con 0x7f230c14bb80 2023-12-13T23:55:33.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.410+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f230c1ef840 con 0x7f230c14bb80 2023-12-13T23:55:33.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.411+0000 7f23027fc640 1 -- 172.21.15.124:0/4043642408 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2308034060 con 0x7f230c14bb80 2023-12-13T23:55:33.409 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:55:33.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 msgr2=0x7f22f006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 0x7f22f006d680 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f23040012e0 tx=0x7f2304007f90 comp rx=0 tx=0).stop 2023-12-13T23:55:33.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 msgr2=0x7f230c1efb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2308009870 tx=0x7f2308004730 comp rx=0 tx=0).stop 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 shutdown_connections 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f22f006b1c0 0x7f22f006d680 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f230c14bb80 0x7f230c1efb20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f230c15dec0 0x7f230c14b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 --2- 172.21.15.124:0/4043642408 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f230c150560 0x7f230c1f1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.412+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 >> 172.21.15.124:0/4043642408 conn(0x7f230c06a4d0 msgr2=0x7f230c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.413+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 shutdown_connections 2023-12-13T23:55:33.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:33.413+0000 7f2313e75640 1 -- 172.21.15.124:0/4043642408 wait complete. 2023-12-13T23:55:33.421 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:55:33.422 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-12-13T23:55:33.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:33.422 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:33.442 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42682sec preferred_lft 42682sec 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2023-12-13T23:55:33.443 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:33.444 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-12-13T23:55:33.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add name ceph-brx type bridge 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr flush dev ceph-brx 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set ceph-brx up 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-12-13T23:55:33.444 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:33.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:33.691 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:55:33.719 INFO:teuthology.orchestra.run.smithi203.stdout:1 2023-12-13T23:55:33.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:33.720 DEBUG:teuthology.orchestra.run.smithi203:> route 2023-12-13T23:55:33.789 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2023-12-13T23:55:33.789 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:55:33.790 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:55:33.791 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:55:33.791 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:55:33.791 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:33.791 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:33.791 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:33.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:55:33.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:55:33.792 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:55:33.792 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:34.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.082 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2023-12-13T23:55:34.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.097 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list-id 2023-12-13T23:55:34.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.152 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:34.152 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:34.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:34.152 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-12-13T23:55:34.152 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:34.238 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-12-13T23:55:34.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-12-13T23:55:34.238 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-12-13T23:55:34.239 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:34.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.344 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:34.344 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:34.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.0 up 2023-12-13T23:55:34.344 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set dev brx.0 master ceph-brx 2023-12-13T23:55:34.344 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:34.387 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2023-12-13T23:55:34.387 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:34.387 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:34.442 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-12-13T23:55:34.442 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:34.442 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:34.497 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2023-12-13T23:55:34.566 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2023-12-13T23:55:34.622 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2023-12-13T23:55:34.623 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:34.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.624 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2023-12-13T23:55:34.676 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:55:34.677 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2023-12-13T23:55:34.719 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2023-12-13T23:55:34.746 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2023-12-13T23:55:34.803 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:34.804 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:34.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.805 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2023-12-13T23:55:34.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:2023-12-13T23:55:34.818+0000 7f9965f67480 -1 init, newargv = 0x55b7c193b860 newargc=15 2023-12-13T23:55:34.816 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[95316]: starting ceph client 2023-12-13T23:55:34.823 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi203.stderr:ceph-fuse[95316]: starting fuse 2023-12-13T23:55:34.832 INFO:teuthology.orchestra.run.smithi203.stdout:41 2023-12-13T23:55:34.832 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> import re 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> import os 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2023-12-13T23:55:34.832 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> return f 2023-12-13T23:55:34.833 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:55:34.834 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:34.834 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.1")) 2023-12-13T23:55:34.834 DEBUG:teuthology.orchestra.run.smithi203:> ' 2023-12-13T23:55:34.951 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.1.95316.asok 2023-12-13T23:55:34.955 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:55:34 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:55:34.962 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.95316.asok 2023-12-13T23:55:34.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:34.962 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.95316.asok status 2023-12-13T23:55:35.099 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "1", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "95316", 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2023-12-13T23:55:35.100 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 4639, 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 4639 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2121645932 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: } 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:55:35.101 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 2121645932 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.4639 192.168.0.1:0/2121645932", 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.1:0/2121645932", 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 6, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 21, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2023-12-13T23:55:35.102 INFO:teuthology.orchestra.run.smithi203.stdout:} 2023-12-13T23:55:35.108 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:55:35.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.223+0000 7f1a44733640 1 Processor -- start 2023-12-13T23:55:35.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.223+0000 7f1a44733640 1 -- start start 2023-12-13T23:55:35.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a44733640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a44733640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 0x7f1a3c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a44733640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a3c0b2ef0 con 0x7f1a3c144440 2023-12-13T23:55:35.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a44733640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b3b70 con 0x7f1a3c144a00 2023-12-13T23:55:35.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a44733640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c0b31d0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a41cce640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 0x7f1a3c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a42cd0640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1a3c144440 0x7f1a3c0b3f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:38238/0 (socket says 172.21.15.124:38238) 2023-12-13T23:55:35.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a42cd0640 1 -- 172.21.15.124:0/2315834378 learned_addr learned my addr 172.21.15.124:0/2315834378 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:35.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.224+0000 7f1a424cf640 1 --2- 172.21.15.124:0/2315834378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a424cf640 1 -- 172.21.15.124:0/2315834378 >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 msgr2=0x7f1a3c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:35.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a424cf640 1 --2- 172.21.15.124:0/2315834378 >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 0x7f1a3c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a424cf640 1 -- 172.21.15.124:0/2315834378 >> v1:172.21.15.203:6789/0 conn(0x7f1a3c144440 legacy=0x7f1a3c0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a424cf640 1 -- 172.21.15.124:0/2315834378 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a3c0b34b0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a41cce640 1 --2- 172.21.15.124:0/2315834378 >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 0x7f1a3c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a424cf640 1 --2- 172.21.15.124:0/2315834378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c141e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a380097a0 tx=0x7f1a3802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5b2acba340cc91a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.225+0000 7f1a414cd640 1 -- 172.21.15.124:0/2315834378 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38031830 con 0x7f1a3c1415f0 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a414cd640 1 -- 172.21.15.124:0/2315834378 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a38031990 con 0x7f1a3c1415f0 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a414cd640 1 -- 172.21.15.124:0/2315834378 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a38031d20 con 0x7f1a3c1415f0 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 -- 172.21.15.124:0/2315834378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 msgr2=0x7f1a3c141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 --2- 172.21.15.124:0/2315834378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c141e00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1a380097a0 tx=0x7f1a3802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 -- 172.21.15.124:0/2315834378 shutdown_connections 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 --2- 172.21.15.124:0/2315834378 >> v2:172.21.15.203:3300/0 conn(0x7f1a3c144a00 0x7f1a3c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 --2- 172.21.15.124:0/2315834378 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c141e00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 -- 172.21.15.124:0/2315834378 >> 172.21.15.124:0/2315834378 conn(0x7f1a3c070e70 msgr2=0x7f1a3c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 -- 172.21.15.124:0/2315834378 shutdown_connections 2023-12-13T23:55:35.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.226+0000 7f1a44733640 1 -- 172.21.15.124:0/2315834378 wait complete. 2023-12-13T23:55:35.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.227+0000 7f1a44733640 1 Processor -- start 2023-12-13T23:55:35.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.227+0000 7f1a44733640 1 -- start start 2023-12-13T23:55:35.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 0x7f1a3c152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c153310 con 0x7f1a3c144a00 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c130230 con 0x7f1a3c144440 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a44733640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a3c130510 con 0x7f1a3c1415f0 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a42cd0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a41cce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 0x7f1a3c152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:51938/0 (socket says 172.21.15.124:51938) 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 -- 172.21.15.124:0/1777716441 learned_addr learned my addr 172.21.15.124:0/1777716441 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a41cce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 0x7f1a3c152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:55122/0 (socket says 172.21.15.124:55122) 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a42cd0640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51290/0 (socket says 172.21.15.124:51290) 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 -- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 msgr2=0x7f1a3c152dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 0x7f1a3c152dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 -- 172.21.15.124:0/1777716441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 msgr2=0x7f1a3c15d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.228+0000 7f1a424cf640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a424cf640 1 -- 172.21.15.124:0/1777716441 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c009400 con 0x7f1a3c1415f0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a42cd0640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a424cf640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a380098c0 tx=0x7f1a380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3803a6f0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a3803a870 con 0x7f1a3c1415f0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a38009400 con 0x7f1a3c1415f0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a3803ac20 con 0x7f1a3c1415f0 2023-12-13T23:55:35.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.229+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a3c130ad0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.230+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3c066760 con 0x7f1a3c1415f0 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.230+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1a38030380 con 0x7f1a3c1415f0 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.231+0000 7f1a32ffd640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 0x7f1a2006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.231+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a380ab380 con 0x7f1a3c1415f0 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.231+0000 7f1a41cce640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 0x7f1a2006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:35.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.231+0000 7f1a41cce640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 0x7f1a2006d680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1540b0 tx=0x7f1a2c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:35.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.235+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1a380761f0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.436+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1a3c0693e0 con 0x7f1a3c1415f0 2023-12-13T23:55:35.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.437+0000 7f1a32ffd640 1 -- 172.21.15.124:0/1777716441 <== mon.2 v2:172.21.15.203:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+73 (secure 0 0 0) 0x7f1a38079810 con 0x7f1a3c1415f0 2023-12-13T23:55:35.435 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:55:35.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 msgr2=0x7f1a2006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 0x7f1a2006d680 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c1540b0 tx=0x7f1a2c009190 comp rx=0 tx=0).stop 2023-12-13T23:55:35.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 msgr2=0x7f1a3c154970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:35.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f1a380098c0 tx=0x7f1a380046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 shutdown_connections 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1a2006b1c0 0x7f1a2006d680 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1a3c144440 0x7f1a3c152dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3c144a00 0x7f1a3c15d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 --2- 172.21.15.124:0/1777716441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1a3c1415f0 0x7f1a3c154970 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 >> 172.21.15.124:0/1777716441 conn(0x7f1a3c070e70 msgr2=0x7f1a3c143ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 shutdown_connections 2023-12-13T23:55:35.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:35.438+0000 7f1a44733640 1 -- 172.21.15.124:0/1777716441 wait complete. 2023-12-13T23:55:35.447 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:55:35.447 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:55:35.447 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi203.front.sepia.ceph.com 2023-12-13T23:55:35.447 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-12-13T23:55:35.447 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:35.448 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:55:35.448 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:55:35.448 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:55:35.448 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-12-13T23:55:35.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.448 DEBUG:teuthology.orchestra.run.smithi203:> ip addr 2023-12-13T23:55:35.464 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:44 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f0 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:d9:19:45 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: altname enp7s0f1 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:22 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:35.465 INFO:teuthology.orchestra.run.smithi203.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 0c:c4:7a:8f:4a:23 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: inet 172.21.15.203/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft 42680sec preferred_lft 42680sec 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::ec4:7aff:fe8f:4a23/64 scope link 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 12:c1:5f:60:93:6f brd ff:ff:ff:ff:ff:ff 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::b84a:a8ff:fe46:6048/64 scope link 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether 12:c1:5f:60:93:6f brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::10c1:5fff:fe60:936f/64 scope link 2023-12-13T23:55:35.466 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:35.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.467 DEBUG:teuthology.orchestra.run.smithi203:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:55:35.533 INFO:teuthology.orchestra.run.smithi203.stdout:1 2023-12-13T23:55:35.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.534 DEBUG:teuthology.orchestra.run.smithi203:> route 2023-12-13T23:55:35.590 INFO:teuthology.orchestra.run.smithi203.stdout:Kernel IP routing table 2023-12-13T23:55:35.591 INFO:teuthology.orchestra.run.smithi203.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:55:35.591 INFO:teuthology.orchestra.run.smithi203.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:55:35.591 INFO:teuthology.orchestra.run.smithi203.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:55:35.591 INFO:teuthology.orchestra.run.smithi203.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:55:35.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.591 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:35.591 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:35.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:55:35.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:55:35.592 DEBUG:teuthology.orchestra.run.smithi203:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:55:35.592 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:35.687 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.687 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2023-12-13T23:55:35.702 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-13T23:55:35.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.703 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list-id 2023-12-13T23:55:35.757 INFO:teuthology.orchestra.run.smithi203.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-12-13T23:55:35.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:35.757 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:35.757 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:35.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:55:35.757 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-12-13T23:55:35.757 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:35.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.111 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:36.145 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether b6:9c:fb:53:df:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::b49c:fbff:fe53:df28/64 scope link 2023-12-13T23:55:36.146 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-12-13T23:55:36.228 INFO:teuthology.orchestra.run.smithi203.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:55:36.228 INFO:teuthology.orchestra.run.smithi203.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:55:36.228 INFO:teuthology.orchestra.run.smithi203.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 ::1/128 scope host 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: link/ether b6:9c:fb:53:df:28 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: inet6 fe80::b49c:fbff:fe53:df28/64 scope link 2023-12-13T23:55:36.229 INFO:teuthology.orchestra.run.smithi203.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:55:36.230 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-12-13T23:55:36.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.230 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:36.230 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:36.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-12-13T23:55:36.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-12-13T23:55:36.231 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-12-13T23:55:36.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-12-13T23:55:36.232 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-12-13T23:55:36.232 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:36.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.358 DEBUG:teuthology.orchestra.run.smithi203:> (cd / && exec bash -c ' 2023-12-13T23:55:36.358 DEBUG:teuthology.orchestra.run.smithi203:> set -e 2023-12-13T23:55:36.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set brx.1 up 2023-12-13T23:55:36.358 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link set dev brx.1 master ceph-brx 2023-12-13T23:55:36.358 DEBUG:teuthology.orchestra.run.smithi203:> ') 2023-12-13T23:55:36.400 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2023-12-13T23:55:36.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:36.400 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:36.460 INFO:teuthology.orchestra.run.smithi203.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-12-13T23:55:36.461 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:55:36.461 DEBUG:teuthology.orchestra.run.smithi203:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:36.516 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2023-12-13T23:55:36.584 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2023-12-13T23:55:36.640 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:36.641 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:55:36.642 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:55:36.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.643 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2023-12-13T23:55:36.695 INFO:teuthology.orchestra.run.smithi203.stdout:41 2023-12-13T23:55:36.695 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2023-12-13T23:55:36.695 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2023-12-13T23:55:36.737 DEBUG:teuthology.orchestra.run.smithi203:> sudo modprobe fuse 2023-12-13T23:55:36.765 DEBUG:teuthology.orchestra.run.smithi203:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:55:36.820 INFO:teuthology.orchestra.run.smithi203.stdout:/proc 2023-12-13T23:55:36.820 INFO:teuthology.orchestra.run.smithi203.stdout:/sys 2023-12-13T23:55:36.820 INFO:teuthology.orchestra.run.smithi203.stdout:/dev 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/security 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/shm 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/pts 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/run 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/cgroup 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/pstore 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/bpf 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/ 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/selinux 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/hugepages 2023-12-13T23:55:36.821 INFO:teuthology.orchestra.run.smithi203.stdout:/dev/mqueue 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/debug 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/tracing 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/fs/fuse/connections 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/sys/kernel/config 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/run/user/1000 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-4 2023-12-13T23:55:36.822 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-5 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-6 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/var/lib/ceph/osd/ceph-7 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:55:36.823 INFO:teuthology.orchestra.run.smithi203.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:55:36.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.824 DEBUG:teuthology.orchestra.run.smithi203:> ls /sys/fs/fuse/connections 2023-12-13T23:55:36.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:2023-12-13T23:55:36.827+0000 7fb773a43480 -1 init, newargv = 0x55767f611ac0 newargc=15 2023-12-13T23:55:36.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:ceph-fuse[95762]: starting ceph client 2023-12-13T23:55:36.829 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi203.stderr:ceph-fuse[95762]: starting fuse 2023-12-13T23:55:36.840 INFO:teuthology.orchestra.run.smithi203.stdout:41 2023-12-13T23:55:36.840 INFO:teuthology.orchestra.run.smithi203.stdout:44 2023-12-13T23:55:36.840 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2023-12-13T23:55:36.840 DEBUG:teuthology.orchestra.run.smithi203:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:55:36.840 DEBUG:teuthology.orchestra.run.smithi203:> import glob 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> import re 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> import os 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> import subprocess 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> def _find_admin_socket(client_name): 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> files = glob.glob(asok_path) 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> # Given a non-glob path, it better be there 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> if "*" not in asok_path: 2023-12-13T23:55:36.841 DEBUG:teuthology.orchestra.run.smithi203:> assert(len(files) == 1) 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> return files[0] 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> for f in files: 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> contents = proc_f.read() 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> if mountpoint in contents: 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> return f 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> 2023-12-13T23:55:36.842 DEBUG:teuthology.orchestra.run.smithi203:> print(_find_admin_socket("client.3")) 2023-12-13T23:55:36.843 DEBUG:teuthology.orchestra.run.smithi203:> ' 2023-12-13T23:55:36.960 INFO:teuthology.orchestra.run.smithi203.stdout:/var/run/ceph/ceph-client.3.95762.asok 2023-12-13T23:55:36.963 INFO:teuthology.orchestra.run.smithi203.stderr:2023-12-13T23:55:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:55:36.971 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.95762.asok 2023-12-13T23:55:36.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:36.971 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.95762.asok status 2023-12-13T23:55:37.106 INFO:teuthology.orchestra.run.smithi203.stdout:{ 2023-12-13T23:55:37.106 INFO:teuthology.orchestra.run.smithi203.stdout: "metadata": { 2023-12-13T23:55:37.106 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:55:37.106 INFO:teuthology.orchestra.run.smithi203.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:55:37.106 INFO:teuthology.orchestra.run.smithi203.stdout: "entity_id": "3", 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "hostname": "smithi203", 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "pid": "95762", 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "root": "/" 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_count": 0, 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "dentry_pinned_count": 0, 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "id": 4505, 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "inst": { 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "name": { 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "client", 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "num": 4505 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2023-12-13T23:55:37.107 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.2:0", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 3291301644 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: } 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": { 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "type": "v1", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "addr": "192.168.0.2:0", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "nonce": 3291301644 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: }, 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "inst_str": "client.4505 192.168.0.2:0/3291301644", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "addr_str": "192.168.0.2:0/3291301644", 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "inode_count": 1, 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "mds_epoch": 6, 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch": 21, 2023-12-13T23:55:37.108 INFO:teuthology.orchestra.run.smithi203.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:55:37.109 INFO:teuthology.orchestra.run.smithi203.stdout: "blocklisted": false, 2023-12-13T23:55:37.109 INFO:teuthology.orchestra.run.smithi203.stdout: "fs_name": "cephfs" 2023-12-13T23:55:37.109 INFO:teuthology.orchestra.run.smithi203.stdout:} 2023-12-13T23:55:37.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.114 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:55:37.115 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:37.133 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:55:37.133 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:37.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.133 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:37.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.207 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:55:37.207 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:37.266 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:55:37.266 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:37.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.266 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:37.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.340 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2023-12-13T23:55:37.340 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:37.355 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2023-12-13T23:55:37.355 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:37.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.356 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:37.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.423 DEBUG:teuthology.orchestra.run.smithi203:> stat --file-system '--printf=%T 2023-12-13T23:55:37.423 DEBUG:teuthology.orchestra.run.smithi203:> ' -- /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:37.478 INFO:teuthology.orchestra.run.smithi203.stdout:fuseblk 2023-12-13T23:55:37.478 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:37.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:37.479 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:37.545 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-12-13T23:55:37.561 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:37.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 Processor -- start 2023-12-13T23:55:37.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 -- start start 2023-12-13T23:55:37.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54c8143490 0x7f54c80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c815e2a0 con 0x7f54c815a4e0 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c815b520 con 0x7f54c8143490 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54cca07640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8143870 con 0x7f54c815e730 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.673+0000 7f54c659c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54c8143490 0x7f54c80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c6d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f54c815a4e0 0x7f54c815a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39714/0 (socket says 172.21.15.124:39714) 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c659c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54c8143490 0x7f54c80b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51298/0 (socket says 172.21.15.124:51298) 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c6d9d640 1 -- 172.21.15.124:0/3239539010 learned_addr learned my addr 172.21.15.124:0/3239539010 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c5d9b640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c659c640 1 -- 172.21.15.124:0/3239539010 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 msgr2=0x7f54c80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c659c640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c659c640 1 -- 172.21.15.124:0/3239539010 >> v1:172.21.15.124:6789/0 conn(0x7f54c815a4e0 legacy=0x7f54c815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c659c640 1 -- 172.21.15.124:0/3239539010 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c815b240 con 0x7f54c8143490 2023-12-13T23:55:37.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.674+0000 7f54c559a640 1 -- 172.21.15.124:0/3239539010 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984125895 0 0) 0x7f54c815e2a0 con 0x7f54c815a4e0 2023-12-13T23:55:37.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54c659c640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c8143490 0x7f54c80b0cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54c000ba40 tx=0x7f54c002e080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7889d7c2e142eb41 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54c559a640 1 -- 172.21.15.124:0/3239539010 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c002ee30 con 0x7f54c8143490 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54c559a640 1 -- 172.21.15.124:0/3239539010 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0036440 con 0x7f54c8143490 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54c559a640 1 -- 172.21.15.124:0/3239539010 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54c00367d0 con 0x7f54c8143490 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 -- 172.21.15.124:0/3239539010 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c8143490 msgr2=0x7f54c80b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c8143490 0x7f54c80b0cf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f54c000ba40 tx=0x7f54c002e080 comp rx=0 tx=0).stop 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 -- 172.21.15.124:0/3239539010 shutdown_connections 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 --2- 172.21.15.124:0/3239539010 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c8143490 0x7f54c80b0cf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 -- 172.21.15.124:0/3239539010 >> 172.21.15.124:0/3239539010 conn(0x7f54c806a4d0 msgr2=0x7f54c806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 -- 172.21.15.124:0/3239539010 shutdown_connections 2023-12-13T23:55:37.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.675+0000 7f54cca07640 1 -- 172.21.15.124:0/3239539010 wait complete. 2023-12-13T23:55:37.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.677+0000 7f54cca07640 1 Processor -- start 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.677+0000 7f54cca07640 1 -- start start 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c8143490 0x7f54c80adea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80ae3e0 con 0x7f54c815a4e0 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8142030 con 0x7f54c8143490 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8141a70 con 0x7f54c815e730 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51306/0 (socket says 172.21.15.124:51306) 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 -- 172.21.15.124:0/2941164383 learned_addr learned my addr 172.21.15.124:0/2941164383 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:37.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c6d9d640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c659c640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c8143490 0x7f54c80adea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 -- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 msgr2=0x7f54c813e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c813e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 -- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c8143490 msgr2=0x7f54c80adea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c8143490 0x7f54c80adea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 -- 172.21.15.124:0/2941164383 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c000b6a0 con 0x7f54c815a4e0 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54c5d9b640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00efc0 tx=0x7f54bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc009b80 con 0x7f54c815a4e0 2023-12-13T23:55:37.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc00f040 con 0x7f54c815a4e0 2023-12-13T23:55:37.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c8141d50 con 0x7f54c815a4e0 2023-12-13T23:55:37.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54bc0146a0 con 0x7f54c815a4e0 2023-12-13T23:55:37.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.678+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c81488e0 con 0x7f54c815a4e0 2023-12-13T23:55:37.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.679+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80b4380 con 0x7f54c815a4e0 2023-12-13T23:55:37.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.680+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f54bc014800 con 0x7f54c815a4e0 2023-12-13T23:55:37.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.680+0000 7f54b6ffd640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 0x7f54a806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.680+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f54bc08a710 con 0x7f54c815a4e0 2023-12-13T23:55:37.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.680+0000 7f54c659c640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 0x7f54a806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.681+0000 7f54c659c640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 0x7f54a806d6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54c0038000 tx=0x7f54c0032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.684 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.685+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f54bc0554f0 con 0x7f54c815a4e0 2023-12-13T23:55:37.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.855+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54c8069400 con 0x7f54c815a4e0 2023-12-13T23:55:37.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.856+0000 7f54b6ffd640 1 -- 172.21.15.124:0/2941164383 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f54bc058b10 con 0x7f54c815a4e0 2023-12-13T23:55:37.855 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:37.855 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:37.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 msgr2=0x7f54a806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 0x7f54a806d6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f54c0038000 tx=0x7f54c0032000 comp rx=0 tx=0).stop 2023-12-13T23:55:37.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 msgr2=0x7f54c8140ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f54bc00efc0 tx=0x7f54bc00c490 comp rx=0 tx=0).stop 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 shutdown_connections 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54a806b1f0 0x7f54a806d6b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c815e730 0x7f54c813e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c815a4e0 0x7f54c8140ff0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 --2- 172.21.15.124:0/2941164383 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c8143490 0x7f54c80adea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 >> 172.21.15.124:0/2941164383 conn(0x7f54c806a4d0 msgr2=0x7f54c80718b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 shutdown_connections 2023-12-13T23:55:37.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.859+0000 7f54cca07640 1 -- 172.21.15.124:0/2941164383 wait complete. 2023-12-13T23:55:37.868 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:37.868 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:37.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.972+0000 7f4b27778640 1 Processor -- start 2023-12-13T23:55:37.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.972+0000 7f4b27778640 1 -- start start 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b27778640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4b20140310 0x7f4b201618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b27778640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4b2006abe0 0x7f4b2006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b27778640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b201406f0 con 0x7f4b201409d0 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b27778640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b20161e20 con 0x7f4b20140310 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b27778640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b20143960 con 0x7f4b2006abe0 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4b20140310 0x7f4b201618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4b20140310 0x7f4b201618e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51322/0 (socket says 172.21.15.124:51322) 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 -- 172.21.15.124:0/2733456740 learned_addr learned my addr 172.21.15.124:0/2733456740 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25d15640 1 --2- 172.21.15.124:0/2733456740 >> v2:172.21.15.203:3300/0 conn(0x7f4b2006abe0 0x7f4b2006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 -- 172.21.15.124:0/2733456740 >> v2:172.21.15.203:3300/0 conn(0x7f4b2006abe0 msgr2=0x7f4b2006efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:37.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 --2- 172.21.15.124:0/2733456740 >> v2:172.21.15.203:3300/0 conn(0x7f4b2006abe0 0x7f4b2006efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 -- 172.21.15.124:0/2733456740 >> v1:172.21.15.203:6789/0 conn(0x7f4b201409d0 legacy=0x7f4b2006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:37.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.973+0000 7f4b25514640 1 -- 172.21.15.124:0/2733456740 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b20142de0 con 0x7f4b20140310 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.974+0000 7f4b25514640 1 --2- 172.21.15.124:0/2733456740 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b201618e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b20154930 tx=0x7f4b1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a29221e0c5966057 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.974+0000 7f4b17fff640 1 -- 172.21.15.124:0/2733456740 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0317e0 con 0x7f4b20140310 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.974+0000 7f4b17fff640 1 -- 172.21.15.124:0/2733456740 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b1c031940 con 0x7f4b20140310 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.974+0000 7f4b17fff640 1 -- 172.21.15.124:0/2733456740 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c031cd0 con 0x7f4b20140310 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 -- 172.21.15.124:0/2733456740 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 msgr2=0x7f4b201618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 --2- 172.21.15.124:0/2733456740 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b201618e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b20154930 tx=0x7f4b1c02b880 comp rx=0 tx=0).stop 2023-12-13T23:55:37.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 -- 172.21.15.124:0/2733456740 shutdown_connections 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 --2- 172.21.15.124:0/2733456740 >> v2:172.21.15.203:3300/0 conn(0x7f4b2006abe0 0x7f4b2006efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 --2- 172.21.15.124:0/2733456740 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b201618e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 -- 172.21.15.124:0/2733456740 >> 172.21.15.124:0/2733456740 conn(0x7f4b200b1e50 msgr2=0x7f4b200b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 -- 172.21.15.124:0/2733456740 shutdown_connections 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.975+0000 7f4b27778640 1 -- 172.21.15.124:0/2733456740 wait complete. 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.976+0000 7f4b27778640 1 Processor -- start 2023-12-13T23:55:37.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.976+0000 7f4b27778640 1 -- start start 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4b2006abe0 0x7f4b20143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b201409d0 0x7f4b201563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b200ad790 con 0x7f4b20140310 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b20146da0 con 0x7f4b201409d0 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2013d0b0 con 0x7f4b2006abe0 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51332/0 (socket says 172.21.15.124:51332) 2023-12-13T23:55:37.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 -- 172.21.15.124:0/1911130828 learned_addr learned my addr 172.21.15.124:0/1911130828 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b25d15640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b201409d0 0x7f4b201563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b25514640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4b2006abe0 0x7f4b20143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 -- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4b2006abe0 msgr2=0x7f4b20143b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4b2006abe0 0x7f4b20143b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 -- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b201409d0 msgr2=0x7f4b201563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b201409d0 0x7f4b201563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 -- 172.21.15.124:0/1911130828 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b1c0094d0 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b24d13640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000c440 tx=0x7f4b1000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b10005480 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b100035f0 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b20146220 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b100039c0 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.977+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b201467e0 con 0x7f4b20140310 2023-12-13T23:55:37.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.978+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b200ad900 con 0x7f4b20140310 2023-12-13T23:55:37.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.979+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4b10003b20 con 0x7f4b20140310 2023-12-13T23:55:37.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.979+0000 7f4b15ffb640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 0x7f4b0406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:37.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.979+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4b10088ec0 con 0x7f4b20140310 2023-12-13T23:55:37.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.979+0000 7f4b25514640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 0x7f4b0406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:37.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.980+0000 7f4b25514640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 0x7f4b0406d6b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:37.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:37.984+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4b200ad900 con 0x7f4b20140310 2023-12-13T23:55:38.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.150+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4b200ad8d0 con 0x7f4b20140310 2023-12-13T23:55:38.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.150+0000 7f4b15ffb640 1 -- 172.21.15.124:0/1911130828 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4b10017070 con 0x7f4b20140310 2023-12-13T23:55:38.150 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 msgr2=0x7f4b0406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 0x7f4b0406d6b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c036000 tx=0x7f4b1c03d040 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 msgr2=0x7f4b2006d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4b1000c440 tx=0x7f4b1000c900 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 shutdown_connections 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4b0406b1f0 0x7f4b0406d6b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4b201409d0 0x7f4b201563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b20140310 0x7f4b2006d720 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 --2- 172.21.15.124:0/1911130828 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4b2006abe0 0x7f4b20143b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 >> 172.21.15.124:0/1911130828 conn(0x7f4b200b1e50 msgr2=0x7f4b20073560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 shutdown_connections 2023-12-13T23:55:38.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.152+0000 7f4b27778640 1 -- 172.21.15.124:0/1911130828 wait complete. 2023-12-13T23:55:38.162 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:38.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.265+0000 7fec71790640 1 Processor -- start 2023-12-13T23:55:38.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.265+0000 7fec71790640 1 -- start start 2023-12-13T23:55:38.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec6c15d330 0x7fec6c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 0x7fec6c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fec6c0d3730 0x7fec6c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6c0b2b10 con 0x7fec6c15d330 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec6c0d3b50 con 0x7fec6c0d3730 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec71790640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c0d1590 con 0x7fec6c0d2910 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec6c15d330 0x7fec6c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec6c15d330 0x7fec6c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51334/0 (socket says 172.21.15.124:51334) 2023-12-13T23:55:38.269 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 -- 172.21.15.124:0/4177381387 learned_addr learned my addr 172.21.15.124:0/4177381387 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6a7fc640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 0x7fec6c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6b7fe640 1 --2- 172.21.15.124:0/4177381387 >> v2:172.21.15.203:3300/0 conn(0x7fec6c0d3730 0x7fec6c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 -- 172.21.15.124:0/4177381387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 msgr2=0x7fec6c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 0x7fec6c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 -- 172.21.15.124:0/4177381387 >> v2:172.21.15.203:3300/0 conn(0x7fec6c0d3730 msgr2=0x7fec6c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 --2- 172.21.15.124:0/4177381387 >> v2:172.21.15.203:3300/0 conn(0x7fec6c0d3730 0x7fec6c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6affd640 1 -- 172.21.15.124:0/4177381387 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0d1870 con 0x7fec6c15d330 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.266+0000 7fec6a7fc640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 0x7fec6c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.267+0000 7fec6affd640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c15d330 0x7fec6c0b8b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0d1510 tx=0x7fec5c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ceb48d9c24e51d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.267+0000 7fec69ffb640 1 -- 172.21.15.124:0/4177381387 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c02c9e0 con 0x7fec6c15d330 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.267+0000 7fec69ffb640 1 -- 172.21.15.124:0/4177381387 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c02cb40 con 0x7fec6c15d330 2023-12-13T23:55:38.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.267+0000 7fec69ffb640 1 -- 172.21.15.124:0/4177381387 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c0343f0 con 0x7fec6c15d330 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 -- 172.21.15.124:0/4177381387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c15d330 msgr2=0x7fec6c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c15d330 0x7fec6c0b8b80 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fec6c0d1510 tx=0x7fec5c02be40 comp rx=0 tx=0).stop 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 -- 172.21.15.124:0/4177381387 shutdown_connections 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 --2- 172.21.15.124:0/4177381387 >> v2:172.21.15.203:3300/0 conn(0x7fec6c0d3730 0x7fec6c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d2910 0x7fec6c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 --2- 172.21.15.124:0/4177381387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c15d330 0x7fec6c0b8b80 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 -- 172.21.15.124:0/4177381387 >> 172.21.15.124:0/4177381387 conn(0x7fec6c072020 msgr2=0x7fec6c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 -- 172.21.15.124:0/4177381387 shutdown_connections 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.268+0000 7fec71790640 1 -- 172.21.15.124:0/4177381387 wait complete. 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.269+0000 7fec71790640 1 Processor -- start 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.269+0000 7fec71790640 1 -- start start 2023-12-13T23:55:38.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d3730 0x7fec6c1f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 0x7fec6c159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c1532f0 con 0x7fec6c0d2910 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c1f3dd0 con 0x7fec6c15d330 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec6c1f4490 con 0x7fec6c0d3730 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51348/0 (socket says 172.21.15.124:51348) 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 -- 172.21.15.124:0/3363888256 learned_addr learned my addr 172.21.15.124:0/3363888256 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6a7fc640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d3730 0x7fec6c1f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6b7fe640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 0x7fec6c159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 -- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d3730 msgr2=0x7fec6c1f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d3730 0x7fec6c1f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 -- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 msgr2=0x7fec6c159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 0x7fec6c159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 -- 172.21.15.124:0/3363888256 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec5c009b30 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6b7fe640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 0x7fec6c159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec6affd640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec5c000f80 tx=0x7fec5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c009510 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec5c03e450 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec6c1f3810 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.270+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec5c03e800 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.271+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec6c1f4120 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.271+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec6c069400 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.272+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fec5c03e960 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.272+0000 7fec537fe640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 0x7fec5806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.272+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec5c0ac700 con 0x7fec6c0d2910 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.272+0000 7fec6a7fc640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 0x7fec5806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.273+0000 7fec6a7fc640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 0x7fec5806d680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec6c072500 tx=0x7fec60008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.276+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fec5c0774c0 con 0x7fec6c0d2910 2023-12-13T23:55:38.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.442+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec6c0d0990 con 0x7fec6c0d2910 2023-12-13T23:55:38.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.442+0000 7fec537fe640 1 -- 172.21.15.124:0/3363888256 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec5c07aae0 con 0x7fec6c0d2910 2023-12-13T23:55:38.442 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 msgr2=0x7fec5806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 0x7fec5806d680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fec6c072500 tx=0x7fec60008040 comp rx=0 tx=0).stop 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 msgr2=0x7fec6c152db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fec5c000f80 tx=0x7fec5c004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 shutdown_connections 2023-12-13T23:55:38.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec5806b1c0 0x7fec5806d680 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec6c15d330 0x7fec6c159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec6c0d3730 0x7fec6c1f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 --2- 172.21.15.124:0/3363888256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec6c0d2910 0x7fec6c152db0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 >> 172.21.15.124:0/3363888256 conn(0x7fec6c072020 msgr2=0x7fec6c0b7430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 shutdown_connections 2023-12-13T23:55:38.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.444+0000 7fec71790640 1 -- 172.21.15.124:0/3363888256 wait complete. 2023-12-13T23:55:38.455 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:38.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.557+0000 7f03f6c77640 1 Processor -- start 2023-12-13T23:55:38.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 -- start start 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f03f00d2910 0x7f03f00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f03f00d3730 0x7f03f00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f00b2b10 con 0x7f03f015d330 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00d3bc0 con 0x7f03f00d3730 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f6c77640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f00d1590 con 0x7f03f00d2910 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f5214640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f03f00d3730 0x7f03f00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f5214640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f03f00d3730 0x7f03f00b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51362/0 (socket says 172.21.15.124:51362) 2023-12-13T23:55:38.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.558+0000 7f03f5214640 1 -- 172.21.15.124:0/934019715 learned_addr learned my addr 172.21.15.124:0/934019715 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03effff640 1 --2- 172.21.15.124:0/934019715 >> v2:172.21.15.203:3300/0 conn(0x7f03f00d2910 0x7f03f00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03f5214640 1 -- 172.21.15.124:0/934019715 >> v2:172.21.15.203:3300/0 conn(0x7f03f00d2910 msgr2=0x7f03f00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03f5214640 1 --2- 172.21.15.124:0/934019715 >> v2:172.21.15.203:3300/0 conn(0x7f03f00d2910 0x7f03f00d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03f5214640 1 -- 172.21.15.124:0/934019715 >> v1:172.21.15.203:6789/0 conn(0x7f03f015d330 legacy=0x7f03f00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03f5214640 1 -- 172.21.15.124:0/934019715 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f00d1870 con 0x7f03f00d3730 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.559+0000 7f03f5214640 1 --2- 172.21.15.124:0/934019715 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f00d3730 0x7f03f00b6390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f03f00d1540 tx=0x7f03e802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0fd6c4377aaead0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03ef7fe640 1 -- 172.21.15.124:0/934019715 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e802c9f0 con 0x7f03f00d3730 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03ef7fe640 1 -- 172.21.15.124:0/934019715 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e802cb50 con 0x7f03f00d3730 2023-12-13T23:55:38.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03ef7fe640 1 -- 172.21.15.124:0/934019715 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e80343f0 con 0x7f03f00d3730 2023-12-13T23:55:38.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 -- 172.21.15.124:0/934019715 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f00d3730 msgr2=0x7f03f00b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 --2- 172.21.15.124:0/934019715 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f00d3730 0x7f03f00b6390 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f03f00d1540 tx=0x7f03e802be50 comp rx=0 tx=0).stop 2023-12-13T23:55:38.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 -- 172.21.15.124:0/934019715 shutdown_connections 2023-12-13T23:55:38.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 --2- 172.21.15.124:0/934019715 >> v2:172.21.15.203:3300/0 conn(0x7f03f00d2910 0x7f03f00d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.565 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 --2- 172.21.15.124:0/934019715 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f00d3730 0x7f03f00b6390 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 -- 172.21.15.124:0/934019715 >> 172.21.15.124:0/934019715 conn(0x7f03f0072020 msgr2=0x7f03f0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 -- 172.21.15.124:0/934019715 shutdown_connections 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.560+0000 7f03f6c77640 1 -- 172.21.15.124:0/934019715 wait complete. 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.561+0000 7f03f6c77640 1 Processor -- start 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.561+0000 7f03f6c77640 1 -- start start 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f00d2910 0x7f03f0158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03f00d3730 0x7f03f0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f01609b0 con 0x7f03f015d330 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00ca4d0 con 0x7f03f00d2910 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f6c77640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f00ca7b0 con 0x7f03f00d3730 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f5214640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f5214640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51368/0 (socket says 172.21.15.124:51368) 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.562+0000 7f03f5214640 1 -- 172.21.15.124:0/2841287411 learned_addr learned my addr 172.21.15.124:0/2841287411 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03effff640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03f00d3730 0x7f03f0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f4a13640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f00d2910 0x7f03f0158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 -- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03f00d3730 msgr2=0x7f03f0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03f00d3730 0x7f03f0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 -- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f00d2910 msgr2=0x7f03f0158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f00d2910 0x7f03f0158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 -- 172.21.15.124:0/2841287411 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e8009b30 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f5214640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009f40 tx=0x7f03e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e803e450 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e803e5d0 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e803e980 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f00caa90 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.563+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f00cb050 con 0x7f03f015d330 2023-12-13T23:55:38.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.564+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f0069400 con 0x7f03f015d330 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.564+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f03e803eae0 con 0x7f03f015d330 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.565+0000 7f03ed7fa640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 0x7f03dc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.565+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f03e80ac970 con 0x7f03f015d330 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.565+0000 7f03f4a13640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 0x7f03dc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.565+0000 7f03f4a13640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 0x7f03dc06d680 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03e0007fd0 tx=0x7f03e0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.570+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f03e8077730 con 0x7f03f015d330 2023-12-13T23:55:38.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.734+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03f00c6810 con 0x7f03f015d330 2023-12-13T23:55:38.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.735+0000 7f03ed7fa640 1 -- 172.21.15.124:0/2841287411 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03e8043020 con 0x7f03f015d330 2023-12-13T23:55:38.734 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:38.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 msgr2=0x7f03dc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 0x7f03dc06d680 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f03e0007fd0 tx=0x7f03e0009040 comp rx=0 tx=0).stop 2023-12-13T23:55:38.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 msgr2=0x7f03f0159370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009f40 tx=0x7f03e8004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 shutdown_connections 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f03dc06b1c0 0x7f03dc06d680 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03f015d330 0x7f03f0159370 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f03f00d3730 0x7f03f0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 --2- 172.21.15.124:0/2841287411 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f03f00d2910 0x7f03f0158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 >> 172.21.15.124:0/2841287411 conn(0x7f03f0072020 msgr2=0x7f03f0072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 shutdown_connections 2023-12-13T23:55:38.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.738+0000 7f03f6c77640 1 -- 172.21.15.124:0/2841287411 wait complete. 2023-12-13T23:55:38.747 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:38.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.849+0000 7fab7c389640 1 Processor -- start 2023-12-13T23:55:38.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.849+0000 7fab7c389640 1 -- start start 2023-12-13T23:55:38.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab7c389640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab7415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab7c389640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fab740b4370 0x7fab740b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab7c389640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab74071770 con 0x7fab740ceb70 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab7c389640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab740b4d20 con 0x7fab740b4370 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab7c389640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab740ccb60 con 0x7fab74144980 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab79924640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fab740ceb70 0x7fab740cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39724/0 (socket says 172.21.15.124:39724) 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab79924640 1 -- 172.21.15.124:0/1421495089 learned_addr learned my addr 172.21.15.124:0/1421495089 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.850+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719148817 0 0) 0x7fab74071770 con 0x7fab740ceb70 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60003520 con 0x7fab740ceb70 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab7415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a926640 1 --2- 172.21.15.124:0/1421495089 >> v2:172.21.15.203:3300/0 conn(0x7fab740b4370 0x7fab740b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2307904935 0 0) 0x7fab60003520 con 0x7fab740ceb70 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 msgr2=0x7fab7415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab7415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 >> v2:172.21.15.203:3300/0 conn(0x7fab740b4370 msgr2=0x7fab740b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 --2- 172.21.15.124:0/1421495089 >> v2:172.21.15.203:3300/0 conn(0x7fab740b4370 0x7fab740b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab740ccdc0 con 0x7fab740ceb70 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fab64001670 con 0x7fab740ceb70 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 >> v1:172.21.15.124:6789/0 conn(0x7fab740ceb70 legacy=0x7fab740cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab60003970 0x7fab60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab600047c0 0x7fab60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab740b4d20 con 0x7fab60003970 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab740ccb60 con 0x7fab600047c0 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000b980 con 0x7fab74144980 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab60003970 0x7fab60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a926640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab79924640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab600047c0 0x7fab60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 -- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 msgr2=0x7fab60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74144980 0x7fab60005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 -- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab600047c0 msgr2=0x7fab60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab600047c0 0x7fab60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.851+0000 7fab7a125640 1 -- 172.21.15.124:0/1421495089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60003520 con 0x7fab60003970 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.852+0000 7fab7a125640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab60003970 0x7fab60003d90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fab70009fc0 tx=0x7fab600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c54a310bfe99719 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.852+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7002cc20 con 0x7fab60003970 2023-12-13T23:55:38.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.852+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7000f040 con 0x7fab60003970 2023-12-13T23:55:38.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.852+0000 7fab79123640 1 -- 172.21.15.124:0/1421495089 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70034440 con 0x7fab60003970 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 -- 172.21.15.124:0/1421495089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab60003970 msgr2=0x7fab60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 --2- 172.21.15.124:0/1421495089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab60003970 0x7fab60003d90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fab70009fc0 tx=0x7fab600057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7a926640 1 -- 172.21.15.124:0/1421495089 reap_dead start 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 -- 172.21.15.124:0/1421495089 shutdown_connections 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 -- 172.21.15.124:0/1421495089 >> 172.21.15.124:0/1421495089 conn(0x7fab7406a4d0 msgr2=0x7fab7406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 -- 172.21.15.124:0/1421495089 shutdown_connections 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.853+0000 7fab7c389640 1 -- 172.21.15.124:0/1421495089 wait complete. 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.854+0000 7fab7c389640 1 Processor -- start 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.854+0000 7fab7c389640 1 -- start start 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.854+0000 7fab7c389640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 0x7fab7415b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 0x7fab74158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab74159400 con 0x7fab7413e6f0 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab741590b0 con 0x7fab60003970 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab74140970 con 0x7fab74140090 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51382/0 (socket says 172.21.15.124:51382) 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 -- 172.21.15.124:0/2601229322 learned_addr learned my addr 172.21.15.124:0/2601229322 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7a125640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 0x7fab7415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7a926640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 0x7fab74158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 -- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 msgr2=0x7fab74158220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 0x7fab74158220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 -- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 msgr2=0x7fab7415b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 0x7fab7415b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 -- 172.21.15.124:0/2601229322 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74140c50 con 0x7fab7413e6f0 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7a125640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 0x7fab7415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:38.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7a926640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 0x7fab74158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab79924640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e700 tx=0x7fab6400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400c6c0 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6400c860 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400cc10 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab74140f30 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.855+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7413eb70 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.856+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab74159540 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.857+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fab6401d460 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.857+0000 7fab6affd640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 0x7fab5806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.857+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab64089860 con 0x7fab7413e6f0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.857+0000 7fab7a125640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 0x7fab5806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:38.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.858+0000 7fab7a125640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 0x7fab5806d6b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab70036000 tx=0x7fab7002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:38.860 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:38.861+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fab74159540 con 0x7fab7413e6f0 2023-12-13T23:55:39.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.028+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab74159540 con 0x7fab7413e6f0 2023-12-13T23:55:39.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.030+0000 7fab6affd640 1 -- 172.21.15.124:0/2601229322 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fab74159540 con 0x7fab7413e6f0 2023-12-13T23:55:39.028 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:39.028 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 msgr2=0x7fab5806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 0x7fab5806d6b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fab70036000 tx=0x7fab7002e000 comp rx=0 tx=0).stop 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 msgr2=0x7fab741f09d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fab6400e700 tx=0x7fab6400ebc0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 shutdown_connections 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab5806b1f0 0x7fab5806d6b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab74140090 0x7fab74158220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab7413e6f0 0x7fab741f09d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 --2- 172.21.15.124:0/2601229322 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab60003970 0x7fab7415b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.031+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 >> 172.21.15.124:0/2601229322 conn(0x7fab7406a4d0 msgr2=0x7fab741576a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.032+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 shutdown_connections 2023-12-13T23:55:39.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.032+0000 7fab7c389640 1 -- 172.21.15.124:0/2601229322 wait complete. 2023-12-13T23:55:39.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:39.041 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:39.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.146+0000 7fe5175a9640 1 Processor -- start 2023-12-13T23:55:39.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.146+0000 7fe5175a9640 1 -- start start 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe5175a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe51015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe5175a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51015e2a0 con 0x7fe51015e730 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe5175a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51015b240 con 0x7fe510143490 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe5175a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51015b520 con 0x7fe5100b1400 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe515b46640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe51015e730 0x7fe51015aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39732/0 (socket says 172.21.15.124:39732) 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe515b46640 1 -- 172.21.15.124:0/3176933471 learned_addr learned my addr 172.21.15.124:0/3176933471 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe514b44640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe51015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283411791 0 0) 0x7fe51015e2a0 con 0x7fe51015e730 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4fc003520 con 0x7fe51015e730 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514473869 0 0) 0x7fe51015b240 con 0x7fe510143490 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe51015e2a0 con 0x7fe510143490 2023-12-13T23:55:39.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3027446434 0 0) 0x7fe4fc003520 con 0x7fe51015e730 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 msgr2=0x7fe51015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe51015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 >> v1:172.21.15.203:6789/0 conn(0x7fe510143490 legacy=0x7fe5100b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51015dc50 con 0x7fe51015e730 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fe508001970 con 0x7fe51015e730 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 >> v1:172.21.15.124:6789/0 conn(0x7fe51015e730 legacy=0x7fe51015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe4fc003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4fc0040e0 0x7fe4fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4fc004d00 0x7fe4fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51015b520 con 0x7fe4fc004d00 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00bac0 con 0x7fe4fc0040e0 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4fc00bc80 con 0x7fe5100b1400 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.147+0000 7fe515b46640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4fc004d00 0x7fe4fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515345640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe4fc003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe514b44640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4fc0040e0 0x7fe4fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 -- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 msgr2=0x7fe4fc003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100b1400 0x7fe4fc003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 -- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4fc0040e0 msgr2=0x7fe4fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4fc0040e0 0x7fe4fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 -- 172.21.15.124:0/3176933471 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4fc003520 con 0x7fe4fc004d00 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.148+0000 7fe515b46640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4fc004d00 0x7fe4fc005180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe50800a680 tx=0x7fe4fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b2d3e9b021dc755 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080072d0 con 0x7fe4fc004d00 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50802f070 con 0x7fe4fc004d00 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe507fff640 1 -- 172.21.15.124:0/3176933471 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5080076b0 con 0x7fe4fc004d00 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 -- 172.21.15.124:0/3176933471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4fc004d00 msgr2=0x7fe4fc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3176933471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4fc004d00 0x7fe4fc005180 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fe50800a680 tx=0x7fe4fc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe515b46640 1 -- 172.21.15.124:0/3176933471 reap_dead start 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 -- 172.21.15.124:0/3176933471 shutdown_connections 2023-12-13T23:55:39.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 -- 172.21.15.124:0/3176933471 >> 172.21.15.124:0/3176933471 conn(0x7fe51006a4d0 msgr2=0x7fe51006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 -- 172.21.15.124:0/3176933471 shutdown_connections 2023-12-13T23:55:39.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.149+0000 7fe5175a9640 1 -- 172.21.15.124:0/3176933471 wait complete. 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.150+0000 7fe5175a9640 1 Processor -- start 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.150+0000 7fe5175a9640 1 -- start start 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe51015b660 0x7fe51015bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100ab100 0x7fe5100ab560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51015c020 con 0x7fe5100ca790 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100aa060 con 0x7fe51015b660 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe5175a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5100aa340 con 0x7fe5100ab100 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51408/0 (socket says 172.21.15.124:51408) 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 -- 172.21.15.124:0/3464020878 learned_addr learned my addr 172.21.15.124:0/3464020878 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 -- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100ab100 msgr2=0x7fe5100ab560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100ab100 0x7fe5100ab560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 -- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe51015b660 msgr2=0x7fe51015bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe51015b660 0x7fe51015bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.151+0000 7fe515345640 1 -- 172.21.15.124:0/3464020878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5100aa620 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe515345640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00f7a0 tx=0x7fe50c00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c00feb0 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe50c005320 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5100aa900 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0072b0 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.152+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5100cab70 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.153+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5100aae20 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.153+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe50c007410 con 0x7fe5100ca790 2023-12-13T23:55:39.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.154+0000 7fe505ffb640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 0x7fe4f406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.154+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe50c0893e0 con 0x7fe5100ca790 2023-12-13T23:55:39.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.154+0000 7fe514b44640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 0x7fe4f406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.154+0000 7fe514b44640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 0x7fe4f406d6b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe51015c1b0 tx=0x7fe5000095b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.158+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe5100aae20 con 0x7fe5100ca790 2023-12-13T23:55:39.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.324+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe5100aae20 con 0x7fe5100ca790 2023-12-13T23:55:39.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.325+0000 7fe505ffb640 1 -- 172.21.15.124:0/3464020878 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5100aae20 con 0x7fe5100ca790 2023-12-13T23:55:39.323 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:39.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 msgr2=0x7fe4f406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 0x7fe4f406d6b0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe51015c1b0 tx=0x7fe5000095b0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 msgr2=0x7fe510132560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00f7a0 tx=0x7fe50c00fc60 comp rx=0 tx=0).stop 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 shutdown_connections 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4f406b1f0 0x7fe4f406d6b0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe51015b660 0x7fe51015bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5100ab100 0x7fe5100ab560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 --2- 172.21.15.124:0/3464020878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5100ca790 0x7fe510132560 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 >> 172.21.15.124:0/3464020878 conn(0x7fe51006a4d0 msgr2=0x7fe5100b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 shutdown_connections 2023-12-13T23:55:39.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.327+0000 7fe5175a9640 1 -- 172.21.15.124:0/3464020878 wait complete. 2023-12-13T23:55:39.336 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:39.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 Processor -- start 2023-12-13T23:55:39.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 -- start start 2023-12-13T23:55:39.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd7d01415f0 0x7fd7d0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d00b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d00b2eb0 con 0x7fd7d0144a00 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d00b3190 con 0x7fd7d01415f0 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.440+0000 7fd7d67fb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d00b3470 con 0x7fd7d006bbf0 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cf7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd7d0144a00 0x7fd7d006b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39736/0 (socket says 172.21.15.124:39736) 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cf7fe640 1 -- 172.21.15.124:0/2092929419 learned_addr learned my addr 172.21.15.124:0/2092929419 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cffff640 1 --2- 172.21.15.124:0/2092929419 >> v2:172.21.15.203:3300/0 conn(0x7fd7d01415f0 0x7fd7d0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7d4d98640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d00b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7ceffd640 1 -- 172.21.15.124:0/2092929419 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154878307 0 0) 0x7fd7d00b2eb0 con 0x7fd7d0144a00 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7ceffd640 1 -- 172.21.15.124:0/2092929419 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b8003520 con 0x7fd7d0144a00 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cffff640 1 -- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 msgr2=0x7fd7d00b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cffff640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d00b3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cffff640 1 -- 172.21.15.124:0/2092929419 >> v1:172.21.15.124:6789/0 conn(0x7fd7d0144a00 legacy=0x7fd7d006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.441+0000 7fd7cffff640 1 -- 172.21.15.124:0/2092929419 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d00b3750 con 0x7fd7d01415f0 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7cffff640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0141e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00097a0 tx=0x7fd7c002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa95864952f2e844 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7ceffd640 1 -- 172.21.15.124:0/2092929419 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0031830 con 0x7fd7d01415f0 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7ceffd640 1 -- 172.21.15.124:0/2092929419 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0031990 con 0x7fd7d01415f0 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7ceffd640 1 -- 172.21.15.124:0/2092929419 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0031d20 con 0x7fd7d01415f0 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 -- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 msgr2=0x7fd7d0141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0141e00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd7c00097a0 tx=0x7fd7c002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 -- 172.21.15.124:0/2092929419 shutdown_connections 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d00b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/2092929419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0141e00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.442+0000 7fd7d67fb640 1 -- 172.21.15.124:0/2092929419 >> 172.21.15.124:0/2092929419 conn(0x7fd7d0070e70 msgr2=0x7fd7d0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.443+0000 7fd7d67fb640 1 -- 172.21.15.124:0/2092929419 shutdown_connections 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.443+0000 7fd7d67fb640 1 -- 172.21.15.124:0/2092929419 wait complete. 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 Processor -- start 2023-12-13T23:55:39.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 -- start start 2023-12-13T23:55:39.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d013cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d0155ef0 con 0x7fd7d0144a00 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d006f0b0 con 0x7fd7d01415f0 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.444+0000 7fd7d67fb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d006eaf0 con 0x7fd7d006bbf0 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51424/0 (socket says 172.21.15.124:51424) 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 -- 172.21.15.124:0/1424426245 learned_addr learned my addr 172.21.15.124:0/1424426245 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7cffff640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d013cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7cf7fe640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 -- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 msgr2=0x7fd7d013cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d013cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 -- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 msgr2=0x7fd7d0155470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0155470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 -- 172.21.15.124:0/1424426245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c0009400 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d4d98640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800efc0 tx=0x7fd7c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c8009b20 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c800f040 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d006edd0 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c80146c0 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.445+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d0158b30 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.446+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d0158d50 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.447+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd7c8014820 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.447+0000 7fd7ccff9640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 0x7fd7bc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.447+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7c808a630 con 0x7fd7d0144a00 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.447+0000 7fd7cffff640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 0x7fd7bc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.448+0000 7fd7cffff640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 0x7fd7bc06d680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0036000 tx=0x7fd7c003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.451+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd7c80554a0 con 0x7fd7d0144a00 2023-12-13T23:55:39.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.617+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd7d0069400 con 0x7fd7d0144a00 2023-12-13T23:55:39.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.618+0000 7fd7ccff9640 1 -- 172.21.15.124:0/1424426245 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd7c8009e60 con 0x7fd7d0144a00 2023-12-13T23:55:39.616 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:39.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.619+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 msgr2=0x7fd7bc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.619+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 0x7fd7bc06d680 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0036000 tx=0x7fd7c003c040 comp rx=0 tx=0).stop 2023-12-13T23:55:39.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.619+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 msgr2=0x7fd7d01559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.619+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd7c800efc0 tx=0x7fd7c800c490 comp rx=0 tx=0).stop 2023-12-13T23:55:39.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 shutdown_connections 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd7bc06b1c0 0x7fd7bc06d680 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd7d01415f0 0x7fd7d0155470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7d0144a00 0x7fd7d01559b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 --2- 172.21.15.124:0/1424426245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd7d006bbf0 0x7fd7d013cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 >> 172.21.15.124:0/1424426245 conn(0x7fd7d0070e70 msgr2=0x7fd7d01424c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 shutdown_connections 2023-12-13T23:55:39.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.620+0000 7fd7d67fb640 1 -- 172.21.15.124:0/1424426245 wait complete. 2023-12-13T23:55:39.629 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:39.732 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.733+0000 7fe4a6061640 1 Processor -- start 2023-12-13T23:55:39.732 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.733+0000 7fe4a6061640 1 -- start start 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe4a6061640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe4a006ea50 0x7fe4a006ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe4a6061640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d0ed0 0x7fe4a00d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe4a6061640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a0161d20 con 0x7fe4a0160fb0 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe4a6061640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a006f370 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe4a6061640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a015e760 con 0x7fe4a006ea50 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d0ed0 0x7fe4a00d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d0ed0 0x7fe4a00d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51432/0 (socket says 172.21.15.124:51432) 2023-12-13T23:55:39.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 -- 172.21.15.124:0/4249595767 learned_addr learned my addr 172.21.15.124:0/4249595767 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49effd640 1 --1- 172.21.15.124:0/4249595767 >> v1:172.21.15.124:6789/0 conn(0x7fe4a0160fb0 0x7fe4a0161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39744/0 (socket says 172.21.15.124:39744) 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/4249595767 >> v2:172.21.15.203:3300/0 conn(0x7fe4a006ea50 0x7fe4a006ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 -- 172.21.15.124:0/4249595767 >> v2:172.21.15.203:3300/0 conn(0x7fe4a006ea50 msgr2=0x7fe4a006ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 --2- 172.21.15.124:0/4249595767 >> v2:172.21.15.203:3300/0 conn(0x7fe4a006ea50 0x7fe4a006ee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 -- 172.21.15.124:0/4249595767 >> v1:172.21.15.124:6789/0 conn(0x7fe4a0160fb0 legacy=0x7fe4a0161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.734+0000 7fe49ffff640 1 -- 172.21.15.124:0/4249595767 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a015d9a0 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.735 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.735+0000 7fe49ffff640 1 --2- 172.21.15.124:0/4249595767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a00d1320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4a006f680 tx=0x7fe49802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97dc4ff1a29ba19 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.735 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.735+0000 7fe49e7fc640 1 -- 172.21.15.124:0/4249595767 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49802ca80 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.735 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.735+0000 7fe49e7fc640 1 -- 172.21.15.124:0/4249595767 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49802cbe0 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.735 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.735+0000 7fe49e7fc640 1 -- 172.21.15.124:0/4249595767 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4980343f0 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 -- 172.21.15.124:0/4249595767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d0ed0 msgr2=0x7fe4a00d1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 --2- 172.21.15.124:0/4249595767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a00d1320 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe4a006f680 tx=0x7fe49802bee0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 -- 172.21.15.124:0/4249595767 shutdown_connections 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 --2- 172.21.15.124:0/4249595767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a00d1320 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 --2- 172.21.15.124:0/4249595767 >> v2:172.21.15.203:3300/0 conn(0x7fe4a006ea50 0x7fe4a006ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 -- 172.21.15.124:0/4249595767 >> 172.21.15.124:0/4249595767 conn(0x7fe4a006a6b0 msgr2=0x7fe4a006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 -- 172.21.15.124:0/4249595767 shutdown_connections 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.736+0000 7fe4a6061640 1 -- 172.21.15.124:0/4249595767 wait complete. 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.737+0000 7fe4a6061640 1 Processor -- start 2023-12-13T23:55:39.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.737+0000 7fe4a6061640 1 -- start start 2023-12-13T23:55:39.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a006ea50 0x7fe4a013d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0156f40 con 0x7fe4a0160fb0 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0159b00 con 0x7fe4a00d0ed0 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe4a6061640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a013dc90 con 0x7fe4a006ea50 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51442/0 (socket says 172.21.15.124:51442) 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 -- 172.21.15.124:0/978969907 learned_addr learned my addr 172.21.15.124:0/978969907 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a006ea50 0x7fe4a013d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49effd640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 -- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a006ea50 msgr2=0x7fe4a013d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a006ea50 0x7fe4a013d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 -- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d0ed0 msgr2=0x7fe4a013f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a013f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.738+0000 7fe49ffff640 1 -- 172.21.15.124:0/978969907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498009b30 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe49ffff640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe498036040 tx=0x7fe498004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498009510 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49803e450 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a013df70 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49803e800 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.739+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a013e530 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.740+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a0070ac0 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.740+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe49802cd50 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.740+0000 7fe487fff640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 0x7fe48c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.740+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4980ac4f0 con 0x7fe4a0160fb0 2023-12-13T23:55:39.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.741+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 0x7fe48c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:39.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.741+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 0x7fe48c06d680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe49000b7c0 tx=0x7fe490008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:39.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.745+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe4980772b0 con 0x7fe4a0160fb0 2023-12-13T23:55:39.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.913+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe4a0069400 con 0x7fe4a0160fb0 2023-12-13T23:55:39.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.913+0000 7fe487fff640 1 -- 172.21.15.124:0/978969907 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe49807a8d0 con 0x7fe4a0160fb0 2023-12-13T23:55:39.912 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 msgr2=0x7fe48c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 0x7fe48c06d680 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe49000b7c0 tx=0x7fe490008040 comp rx=0 tx=0).stop 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 msgr2=0x7fe4a0156a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe498036040 tx=0x7fe498004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 shutdown_connections 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48c06b1c0 0x7fe48c06d680 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d0ed0 0x7fe4a013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a0160fb0 0x7fe4a0156a00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 --2- 172.21.15.124:0/978969907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a006ea50 0x7fe4a013d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:39.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 >> 172.21.15.124:0/978969907 conn(0x7fe4a006a6b0 msgr2=0x7fe4a006aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:39.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 shutdown_connections 2023-12-13T23:55:39.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:39.915+0000 7fe4a6061640 1 -- 172.21.15.124:0/978969907 wait complete. 2023-12-13T23:55:39.924 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:40.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.027+0000 7f3aef7b6640 1 Processor -- start 2023-12-13T23:55:40.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.027+0000 7f3aef7b6640 1 -- start start 2023-12-13T23:55:40.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aef7b6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1400 0x7f3ae815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aef7b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ae815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aef7b6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ae815e2a0 con 0x7f3ae8143490 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aef7b6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae815b520 con 0x7f3ae80b1400 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aef7b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8143870 con 0x7f3ae815e730 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aed552640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3ae8143490 0x7f3ae80b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39754/0 (socket says 172.21.15.124:39754) 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aed552640 1 -- 172.21.15.124:0/2889489227 learned_addr learned my addr 172.21.15.124:0/2889489227 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134784145 0 0) 0x7f3ae815e2a0 con 0x7f3ae8143490 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aedd53640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ae815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad4005730 con 0x7f3ae8143490 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2889489227 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1400 0x7f3ae815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1682424508 0 0) 0x7f3ad4005730 con 0x7f3ae8143490 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 msgr2=0x7f3ae815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ae815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1400 msgr2=0x7f3ae815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 --2- 172.21.15.124:0/2889489227 >> v2:172.21.15.203:3300/0 conn(0x7f3ae80b1400 0x7f3ae815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae815b240 con 0x7f3ae8143490 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f3ae4001670 con 0x7f3ae8143490 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 >> v1:172.21.15.124:6789/0 conn(0x7f3ae8143490 legacy=0x7f3ae80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ad4005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ad4006100 0x7f3ad40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ad4006f10 0x7f3ad4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.028+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae815b520 con 0x7f3ad4006100 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae8143870 con 0x7f3ad4006f10 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad400db90 con 0x7f3ae815e730 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ad4006100 0x7f3ad40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aed552640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ad4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aedd53640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ad4006f10 0x7f3ad4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 -- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 msgr2=0x7f3ad4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ad4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 -- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ad4006f10 msgr2=0x7f3ad4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ad4006f10 0x7f3ad4007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aecd51640 1 -- 172.21.15.124:0/2889489227 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad4005730 con 0x7f3ad4006100 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.029+0000 7f3aed552640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ae815e730 0x7f3ad4005b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ad4006100 0x7f3ad40069d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009ac0 tx=0x7f3ad4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=852b69d0b5b49be9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad802c6d0 con 0x7f3ad4006100 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad802c830 con 0x7f3ad4006100 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3adffff640 1 -- 172.21.15.124:0/2889489227 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad802cbc0 con 0x7f3ad4006100 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2889489227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ad4006100 msgr2=0x7f3ad40069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2889489227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ad4006100 0x7f3ad40069d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009ac0 tx=0x7f3ad4007a00 comp rx=0 tx=0).stop 2023-12-13T23:55:40.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aedd53640 1 -- 172.21.15.124:0/2889489227 reap_dead start 2023-12-13T23:55:40.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2889489227 shutdown_connections 2023-12-13T23:55:40.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2889489227 >> 172.21.15.124:0/2889489227 conn(0x7f3ae806a4d0 msgr2=0x7f3ae806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2889489227 shutdown_connections 2023-12-13T23:55:40.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.030+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2889489227 wait complete. 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.031+0000 7f3aef7b6640 1 Processor -- start 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 -- start start 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 0x7f3ae81eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae81598b0 0x7f3ae81f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae813dda0 con 0x7f3ae80a84d0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae81f21f0 con 0x7f3ae81598b0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aef7b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae813d190 con 0x7f3ad4006100 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aedd53640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aedd53640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51456/0 (socket says 172.21.15.124:51456) 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aedd53640 1 -- 172.21.15.124:0/2270266563 learned_addr learned my addr 172.21.15.124:0/2270266563 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.032+0000 7f3aed552640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 0x7f3ae81eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aecd51640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae81598b0 0x7f3ae81f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 -- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 msgr2=0x7f3ae81eeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 0x7f3ae81eeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 -- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae81598b0 msgr2=0x7f3ae81f1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae81598b0 0x7f3ae81f1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 -- 172.21.15.124:0/2270266563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae813d470 con 0x7f3ae80a84d0 2023-12-13T23:55:40.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aed552640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 0x7f3ae81eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aedd53640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80ccfb0 tx=0x7f3ae00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae000e070 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae0011e10 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae813d750 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0003990 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.033+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae80a8950 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.034+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae8069400 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.034+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3ae0003af0 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.035+0000 7f3addffb640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 0x7f3acc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.035+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ae0089fe0 con 0x7f3ae80a84d0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.035+0000 7f3aed552640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 0x7f3acc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.036+0000 7f3aed552640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 0x7f3acc06d680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000fb0 tx=0x7f3ae4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.039+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3ae0054d10 con 0x7f3ae80a84d0 2023-12-13T23:55:40.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.208+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ae815af80 con 0x7f3ae80a84d0 2023-12-13T23:55:40.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.210+0000 7f3addffb640 1 -- 172.21.15.124:0/2270266563 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f3ae0058330 con 0x7f3ae80a84d0 2023-12-13T23:55:40.208 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:40.208 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:40.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.211+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 msgr2=0x7f3acc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.211+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 0x7f3acc06d680 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4000fb0 tx=0x7f3ae4009040 comp rx=0 tx=0).stop 2023-12-13T23:55:40.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.211+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 msgr2=0x7f3ae813e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.211+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80ccfb0 tx=0x7f3ae00115a0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 shutdown_connections 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3acc06b1c0 0x7f3acc06d680 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3ae81598b0 0x7f3ae81f1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3ae80a84d0 0x7f3ae813e230 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 --2- 172.21.15.124:0/2270266563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3ad4006100 0x7f3ae81eeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 >> 172.21.15.124:0/2270266563 conn(0x7f3ae806a4d0 msgr2=0x7f3ae80ce7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 shutdown_connections 2023-12-13T23:55:40.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.212+0000 7f3aef7b6640 1 -- 172.21.15.124:0/2270266563 wait complete. 2023-12-13T23:55:40.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:40.221 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:40.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.325+0000 7fb9a3f73640 1 Processor -- start 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.325+0000 7fb9a3f73640 1 -- start start 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a3f73640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c0ae480 0x7fb99c06d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a3f73640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb99c072390 0x7fb99c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a3f73640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb99c0ae910 con 0x7fb99c15d2b0 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a3f73640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99c0b2200 con 0x7fb99c072390 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a3f73640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0b24e0 con 0x7fb99c0ae480 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a1d0f640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb99c15d2b0 0x7fb99c071c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:39766/0 (socket says 172.21.15.124:39766) 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a1d0f640 1 -- 172.21.15.124:0/4255367536 learned_addr learned my addr 172.21.15.124:0/4255367536 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457671560 0 0) 0x7fb99c0ae910 con 0x7fb99c15d2b0 2023-12-13T23:55:40.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a2510640 1 --2- 172.21.15.124:0/4255367536 >> v2:172.21.15.203:3300/0 conn(0x7fb99c072390 0x7fb99c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a150e640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c0ae480 0x7fb99c06d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.326+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb988003520 con 0x7fb99c15d2b0 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749390575 0 0) 0x7fb988003520 con 0x7fb99c15d2b0 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c0ae480 msgr2=0x7fb99c06d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c0ae480 0x7fb99c06d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 >> v2:172.21.15.203:3300/0 conn(0x7fb99c072390 msgr2=0x7fb99c06d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 --2- 172.21.15.124:0/4255367536 >> v2:172.21.15.203:3300/0 conn(0x7fb99c072390 0x7fb99c06d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0b40a0 con 0x7fb99c15d2b0 2023-12-13T23:55:40.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fb9980019f0 con 0x7fb99c15d2b0 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 >> v1:172.21.15.124:6789/0 conn(0x7fb99c15d2b0 legacy=0x7fb99c071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb988003970 0x7fb988003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880042d0 0x7fb9880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb988004e10 0x7fb988005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0b2200 con 0x7fb988004e10 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c0b24e0 con 0x7fb9880042d0 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98800bda0 con 0x7fb988003970 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb988004e10 0x7fb988005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a1d0f640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb988003970 0x7fb988003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a150e640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880042d0 0x7fb9880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 -- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb988003970 msgr2=0x7fb988003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb988003970 0x7fb988003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 -- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880042d0 msgr2=0x7fb9880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9880042d0 0x7fb9880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.327+0000 7fb9a2510640 1 -- 172.21.15.124:0/4255367536 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb988003520 con 0x7fb988004e10 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.328+0000 7fb9a2510640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb988004e10 0x7fb988005290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb99c071650 tx=0x7fb9880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95522ec537b2fa4d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.328+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99402c680 con 0x7fb988004e10 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.328+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99402c7e0 con 0x7fb988004e10 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.328+0000 7fb9a0d0d640 1 -- 172.21.15.124:0/4255367536 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99402cb90 con 0x7fb988004e10 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 -- 172.21.15.124:0/4255367536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb988004e10 msgr2=0x7fb988005290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/4255367536 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb988004e10 0x7fb988005290 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb99c071650 tx=0x7fb9880057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a2510640 1 -- 172.21.15.124:0/4255367536 reap_dead start 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 -- 172.21.15.124:0/4255367536 shutdown_connections 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 -- 172.21.15.124:0/4255367536 >> 172.21.15.124:0/4255367536 conn(0x7fb99c0b4720 msgr2=0x7fb99c0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 -- 172.21.15.124:0/4255367536 shutdown_connections 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.329+0000 7fb9a3f73640 1 -- 172.21.15.124:0/4255367536 wait complete. 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.330+0000 7fb9a3f73640 1 Processor -- start 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.330+0000 7fb9a3f73640 1 -- start start 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c150f10 0x7fb99c06d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c06db60 0x7fb99c152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c06dfe0 con 0x7fb99c1f23e0 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c1f27c0 con 0x7fb99c150f10 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99c151960 con 0x7fb99c06db60 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:51480/0 (socket says 172.21.15.124:51480) 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 -- 172.21.15.124:0/1004399461 learned_addr learned my addr 172.21.15.124:0/1004399461 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a2510640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c06db60 0x7fb99c152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a150e640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c150f10 0x7fb99c06d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 -- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c06db60 msgr2=0x7fb99c152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c06db60 0x7fb99c152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 -- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c150f10 msgr2=0x7fb99c06d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c150f10 0x7fb99c06d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 -- 172.21.15.124:0/1004399461 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c0b2af0 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a1d0f640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb99800a3e0 tx=0x7fb99800a7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998013600 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb998013780 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb99c151680 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb998013b30 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.331+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb99c15e7a0 con 0x7fb99c1f23e0 2023-12-13T23:55:40.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.332+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb99c15e310 con 0x7fb99c1f23e0 2023-12-13T23:55:40.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.333+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb998013c90 con 0x7fb99c1f23e0 2023-12-13T23:55:40.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.333+0000 7fb9927fc640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 0x7fb98006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.333+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb99808aa40 con 0x7fb99c1f23e0 2023-12-13T23:55:40.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.333+0000 7fb9a150e640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 0x7fb98006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.334+0000 7fb9a150e640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 0x7fb98006d6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0b1ec0 tx=0x7fb98c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.337+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb99808ce60 con 0x7fb99c1f23e0 2023-12-13T23:55:40.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.505+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb99c069400 con 0x7fb99c1f23e0 2023-12-13T23:55:40.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.505+0000 7fb9927fc640 1 -- 172.21.15.124:0/1004399461 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb998017070 con 0x7fb99c1f23e0 2023-12-13T23:55:40.504 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:40.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 msgr2=0x7fb98006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 0x7fb98006d6b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb99c0b1ec0 tx=0x7fb98c009190 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 msgr2=0x7fb99c14eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb99800a3e0 tx=0x7fb99800a7e0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 shutdown_connections 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb98006b1f0 0x7fb98006d6b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb99c150f10 0x7fb99c06d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb99c06db60 0x7fb99c152bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 --2- 172.21.15.124:0/1004399461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb99c1f23e0 0x7fb99c14eaf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 >> 172.21.15.124:0/1004399461 conn(0x7fb99c0b4720 msgr2=0x7fb99c0ace90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 shutdown_connections 2023-12-13T23:55:40.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.507+0000 7fb9a3f73640 1 -- 172.21.15.124:0/1004399461 wait complete. 2023-12-13T23:55:40.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:40.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.620+0000 7ff97f01d640 1 Processor -- start 2023-12-13T23:55:40.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.620+0000 7ff97f01d640 1 -- start start 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff97f01d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff978140340 0x7ff978166000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff97f01d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff97815ba40 con 0x7ff97806f210 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff97f01d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9781416e0 con 0x7ff97815c2f0 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff97f01d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff97815c6d0 con 0x7ff978140340 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff977fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ff97806f210 0x7ff978140800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56778/0 (socket says 172.21.15.124:56778) 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff97d5ba640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff978140340 0x7ff978166000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.621+0000 7ff977fff640 1 -- 172.21.15.124:0/1124605688 learned_addr learned my addr 172.21.15.124:0/1124605688 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:40.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.622+0000 7ff97d5ba640 1 -- 172.21.15.124:0/1124605688 >> v1:172.21.15.203:6789/0 conn(0x7ff97815c2f0 legacy=0x7ff97806eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.622+0000 7ff97d5ba640 1 -- 172.21.15.124:0/1124605688 >> v1:172.21.15.124:6789/0 conn(0x7ff97806f210 legacy=0x7ff978140800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.622+0000 7ff97d5ba640 1 -- 172.21.15.124:0/1124605688 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff978141400 con 0x7ff978140340 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.622+0000 7ff97d5ba640 1 --2- 172.21.15.124:0/1124605688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff978140340 0x7ff978166000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff9781418d0 tx=0x7ff97002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05334972bf9ce6a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.622+0000 7ff9777fe640 1 -- 172.21.15.124:0/1124605688 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97002cc50 con 0x7ff978140340 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff9777fe640 1 -- 172.21.15.124:0/1124605688 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97002cdb0 con 0x7ff978140340 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff9777fe640 1 -- 172.21.15.124:0/1124605688 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970034660 con 0x7ff978140340 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 -- 172.21.15.124:0/1124605688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff978140340 msgr2=0x7ff978166000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 --2- 172.21.15.124:0/1124605688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff978140340 0x7ff978166000 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff9781418d0 tx=0x7ff97002bea0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 -- 172.21.15.124:0/1124605688 shutdown_connections 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 --2- 172.21.15.124:0/1124605688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff978140340 0x7ff978166000 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 -- 172.21.15.124:0/1124605688 >> 172.21.15.124:0/1124605688 conn(0x7ff97806a6b0 msgr2=0x7ff97806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 -- 172.21.15.124:0/1124605688 shutdown_connections 2023-12-13T23:55:40.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.623+0000 7ff97f01d640 1 -- 172.21.15.124:0/1124605688 wait complete. 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.624+0000 7ff97f01d640 1 Processor -- start 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.624+0000 7ff97f01d640 1 -- start start 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff978140340 0x7ff978151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff97815c2f0 0x7ff9780b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97813b680 con 0x7ff97806f210 2023-12-13T23:55:40.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff978139a40 con 0x7ff97815c2f0 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97f01d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff978151f60 con 0x7ff978140340 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39736/0 (socket says 172.21.15.124:39736) 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 -- 172.21.15.124:0/3629193859 learned_addr learned my addr 172.21.15.124:0/3629193859 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97d5ba640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff97815c2f0 0x7ff9780b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff977fff640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff978140340 0x7ff978151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 -- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff978140340 msgr2=0x7ff978151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff978140340 0x7ff978151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 -- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff97815c2f0 msgr2=0x7ff9780b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.625+0000 7ff97cdb9640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff97815c2f0 0x7ff9780b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff97cdb9640 1 -- 172.21.15.124:0/3629193859 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff970009a70 con 0x7ff97806f210 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff97cdb9640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff96800b3e0 tx=0x7ff96800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968011830 con 0x7ff97806f210 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9680119b0 con 0x7ff97806f210 2023-12-13T23:55:40.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff978139d00 con 0x7ff97806f210 2023-12-13T23:55:40.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff968011d60 con 0x7ff97806f210 2023-12-13T23:55:40.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.626+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97813a2c0 con 0x7ff97806f210 2023-12-13T23:55:40.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.627+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff978066e90 con 0x7ff97806f210 2023-12-13T23:55:40.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.627+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff96801a400 con 0x7ff97806f210 2023-12-13T23:55:40.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.628+0000 7ff9757fa640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 0x7ff96406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.628+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff968089be0 con 0x7ff97806f210 2023-12-13T23:55:40.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.628+0000 7ff977fff640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 0x7ff96406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.628+0000 7ff977fff640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 0x7ff96406d680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff9781382c0 tx=0x7ff96c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.632+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff968054910 con 0x7ff97806f210 2023-12-13T23:55:40.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.800+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff978069400 con 0x7ff97806f210 2023-12-13T23:55:40.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.801+0000 7ff9757fa640 1 -- 172.21.15.124:0/3629193859 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff9680150a0 con 0x7ff97806f210 2023-12-13T23:55:40.799 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:40.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 msgr2=0x7ff96406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 0x7ff96406d680 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff9781382c0 tx=0x7ff96c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:40.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 msgr2=0x7ff978139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff96800b3e0 tx=0x7ff96800b8a0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 shutdown_connections 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff96406b1c0 0x7ff96406d680 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff97815c2f0 0x7ff9780b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff978140340 0x7ff978151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 --2- 172.21.15.124:0/3629193859 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff97806f210 0x7ff978139500 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 >> 172.21.15.124:0/3629193859 conn(0x7ff97806a6b0 msgr2=0x7ff97806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 shutdown_connections 2023-12-13T23:55:40.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.803+0000 7ff97f01d640 1 -- 172.21.15.124:0/3629193859 wait complete. 2023-12-13T23:55:40.812 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:40.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.916+0000 7f956613c640 1 Processor -- start 2023-12-13T23:55:40.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.916+0000 7f956613c640 1 -- start start 2023-12-13T23:55:40.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f956613c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9560155790 0x7f956006f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f956613c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956015ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f956613c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9560155b70 con 0x7f956015b5e0 2023-12-13T23:55:40.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f956613c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f956006ee30 con 0x7f9560155790 2023-12-13T23:55:40.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f956613c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956006f3b0 con 0x7f956006abe0 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9560155790 0x7f956006f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9560155790 0x7f956006f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39740/0 (socket says 172.21.15.124:39740) 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 -- 172.21.15.124:0/2508096122 learned_addr learned my addr 172.21.15.124:0/2508096122 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955ffff640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956015ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 -- 172.21.15.124:0/2508096122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 msgr2=0x7f956015ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956015ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 -- 172.21.15.124:0/2508096122 >> v1:172.21.15.203:6789/0 conn(0x7f956015b5e0 legacy=0x7f956006a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.917+0000 7f955f7fe640 1 -- 172.21.15.124:0/2508096122 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956015c170 con 0x7f9560155790 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.918+0000 7f955f7fe640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956006f840 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9550009870 tx=0x7f955002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=738615058185a2d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.918+0000 7f955e7fc640 1 -- 172.21.15.124:0/2508096122 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550031640 con 0x7f9560155790 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.918+0000 7f955e7fc640 1 -- 172.21.15.124:0/2508096122 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95500317a0 con 0x7f9560155790 2023-12-13T23:55:40.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.918+0000 7f955e7fc640 1 -- 172.21.15.124:0/2508096122 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9550031b30 con 0x7f9560155790 2023-12-13T23:55:40.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 -- 172.21.15.124:0/2508096122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 msgr2=0x7f956006f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956006f840 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9550009870 tx=0x7f955002b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 -- 172.21.15.124:0/2508096122 shutdown_connections 2023-12-13T23:55:40.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956015ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 --2- 172.21.15.124:0/2508096122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956006f840 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 -- 172.21.15.124:0/2508096122 >> 172.21.15.124:0/2508096122 conn(0x7f95600b1e50 msgr2=0x7f95600b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 -- 172.21.15.124:0/2508096122 shutdown_connections 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.919+0000 7f956613c640 1 -- 172.21.15.124:0/2508096122 wait complete. 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.920+0000 7f956613c640 1 Processor -- start 2023-12-13T23:55:40.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.920+0000 7f956613c640 1 -- start start 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 0x7f9560155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956014c330 con 0x7f9560155790 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956014e300 con 0x7f956015b5e0 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956014e560 con 0x7f956006abe0 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39746/0 (socket says 172.21.15.124:39746) 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 -- 172.21.15.124:0/3873781636 learned_addr learned my addr 172.21.15.124:0/3873781636 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955f7fe640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955ffff640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 0x7f9560155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 -- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 msgr2=0x7f956014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956014d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 -- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 msgr2=0x7f9560155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 0x7f9560155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 -- 172.21.15.124:0/3873781636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95500094d0 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955f7fe640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955ffff640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 0x7f9560155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f955effd640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f955400ba50 tx=0x7f955400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f955400cbe0 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f955400cd80 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956014e840 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9554014680 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.921+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956014ee00 con 0x7f9560155790 2023-12-13T23:55:40.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.922+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9560069400 con 0x7f9560155790 2023-12-13T23:55:40.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.923+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f95540147e0 con 0x7f9560155790 2023-12-13T23:55:40.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.923+0000 7f9547fff640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 0x7f954c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:40.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.923+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9554089880 con 0x7f9560155790 2023-12-13T23:55:40.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.924+0000 7f955f7fe640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 0x7f954c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:40.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.924+0000 7f955f7fe640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 0x7f954c06d680 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9550036000 tx=0x7f955003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:40.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:40.928+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9554054660 con 0x7f9560155790 2023-12-13T23:55:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.095+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f95600b2c50 con 0x7f9560155790 2023-12-13T23:55:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.095+0000 7f9547fff640 1 -- 172.21.15.124:0/3873781636 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f95540190a0 con 0x7f9560155790 2023-12-13T23:55:41.093 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 msgr2=0x7f954c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 0x7f954c06d680 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9550036000 tx=0x7f955003c040 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 msgr2=0x7f956014ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f955400ba50 tx=0x7f955400bf10 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 shutdown_connections 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f954c06b1c0 0x7f954c06d680 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f956015b5e0 0x7f9560155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9560155790 0x7f956014ddc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 --2- 172.21.15.124:0/3873781636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f956006abe0 0x7f956014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 >> 172.21.15.124:0/3873781636 conn(0x7f95600b1e50 msgr2=0x7f95600714a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 shutdown_connections 2023-12-13T23:55:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.097+0000 7f956613c640 1 -- 172.21.15.124:0/3873781636 wait complete. 2023-12-13T23:55:41.107 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:41.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.209+0000 7fe4a567c640 1 Processor -- start 2023-12-13T23:55:41.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.209+0000 7fe4a567c640 1 -- start start 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe4a015d330 0x7fe4a00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d2910 0x7fe4a00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d3730 0x7fe4a00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a00b2b10 con 0x7fe4a00d3730 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a00d3b50 con 0x7fe4a015d330 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe4a567c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a00d1590 con 0x7fe4a00d2910 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d3730 0x7fe4a00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4a00d3730 0x7fe4a00b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39756/0 (socket says 172.21.15.124:39756) 2023-12-13T23:55:41.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2252934096 learned_addr learned my addr 172.21.15.124:0/2252934096 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49effd640 1 --2- 172.21.15.124:0/2252934096 >> v2:172.21.15.203:3300/0 conn(0x7fe4a015d330 0x7fe4a00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe497fff640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d2910 0x7fe4a00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2252934096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d2910 msgr2=0x7fe4a00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d2910 0x7fe4a00d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2252934096 >> v2:172.21.15.203:3300/0 conn(0x7fe4a015d330 msgr2=0x7fe4a00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2252934096 >> v2:172.21.15.203:3300/0 conn(0x7fe4a015d330 0x7fe4a00b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.210+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2252934096 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a00d1870 con 0x7fe4a00d3730 2023-12-13T23:55:41.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d3730 0x7fe4a00b6390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00d0d10 tx=0x7fe49802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55e000e6733794d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe49e7fc640 1 -- 172.21.15.124:0/2252934096 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49802c9e0 con 0x7fe4a00d3730 2023-12-13T23:55:41.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe49e7fc640 1 -- 172.21.15.124:0/2252934096 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49802cb40 con 0x7fe4a00d3730 2023-12-13T23:55:41.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe49e7fc640 1 -- 172.21.15.124:0/2252934096 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4980343f0 con 0x7fe4a00d3730 2023-12-13T23:55:41.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe4a567c640 1 -- 172.21.15.124:0/2252934096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d3730 msgr2=0x7fe4a00b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.211+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d3730 0x7fe4a00b6390 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00d0d10 tx=0x7fe49802be40 comp rx=0 tx=0).stop 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 -- 172.21.15.124:0/2252934096 shutdown_connections 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a00d3730 0x7fe4a00b6390 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2252934096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d2910 0x7fe4a00d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2252934096 >> v2:172.21.15.203:3300/0 conn(0x7fe4a015d330 0x7fe4a00b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 -- 172.21.15.124:0/2252934096 >> 172.21.15.124:0/2252934096 conn(0x7fe4a0072020 msgr2=0x7fe4a0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 -- 172.21.15.124:0/2252934096 shutdown_connections 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.212+0000 7fe4a567c640 1 -- 172.21.15.124:0/2252934096 wait complete. 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 Processor -- start 2023-12-13T23:55:41.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 -- start start 2023-12-13T23:55:41.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 0x7fe4a015d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 0x7fe4a015dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a01433f0 con 0x7fe4a015d330 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0145db0 con 0x7fe4a00d2910 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.213+0000 7fe4a567c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a0146090 con 0x7fe4a00d3730 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39764/0 (socket says 172.21.15.124:39764) 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2106778688 learned_addr learned my addr 172.21.15.124:0/2106778688 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe497fff640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 0x7fe4a015dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49effd640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 0x7fe4a015d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 msgr2=0x7fe4a015dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 0x7fe4a015dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 msgr2=0x7fe4a015d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 0x7fe4a015d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 -- 172.21.15.124:0/2106778688 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498009b30 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe497fff640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 0x7fe4a015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49effd640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 0x7fe4a015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe49f7fe640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe498009f40 tx=0x7fe498004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498009510 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49803e450 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a0146370 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49803e7e0 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.214+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a00cb830 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.215+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a0069400 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.216+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe49803e940 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.216+0000 7fe4977fe640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 0x7fe48006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.216+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4980ac810 con 0x7fe4a015d330 2023-12-13T23:55:41.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.216+0000 7fe49effd640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 0x7fe48006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.217+0000 7fe49effd640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 0x7fe48006d680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe490006fd0 tx=0x7fe490008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.220+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe498077660 con 0x7fe4a015d330 2023-12-13T23:55:41.391 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.392+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4a0072740 con 0x7fe4a015d330 2023-12-13T23:55:41.392 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.394+0000 7fe4977fe640 1 -- 172.21.15.124:0/2106778688 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7fe49807ac80 con 0x7fe4a015d330 2023-12-13T23:55:41.392 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:41.392 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:41.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.395+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 msgr2=0x7fe48006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.395+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 0x7fe48006d680 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe490006fd0 tx=0x7fe490008040 comp rx=0 tx=0).stop 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.395+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 msgr2=0x7fe4a0144d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.395+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe498009f40 tx=0x7fe498004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 shutdown_connections 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe48006b1c0 0x7fe48006d680 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4a015d330 0x7fe4a0144d60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4a00d3730 0x7fe4a015dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 --2- 172.21.15.124:0/2106778688 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4a00d2910 0x7fe4a015d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 >> 172.21.15.124:0/2106778688 conn(0x7fe4a0072020 msgr2=0x7fe4a00b8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 shutdown_connections 2023-12-13T23:55:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.396+0000 7fe4a567c640 1 -- 172.21.15.124:0/2106778688 wait complete. 2023-12-13T23:55:41.405 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:41.406 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:41.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.509+0000 7f6e4b4bb640 1 Processor -- start 2023-12-13T23:55:41.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.510+0000 7f6e4b4bb640 1 -- start start 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.510+0000 7f6e4b4bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e4415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.510+0000 7f6e4b4bb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4415e2a0 con 0x7f6e4415e730 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.510+0000 7f6e4b4bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e4415b240 con 0x7f6e44143490 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.510+0000 7f6e4b4bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4415b520 con 0x7f6e440b1400 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49a58640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6e4415e730 0x7f6e4415aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56786/0 (socket says 172.21.15.124:56786) 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49a58640 1 -- 172.21.15.124:0/1160450691 learned_addr learned my addr 172.21.15.124:0/1160450691 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e48a56640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e4415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381986106 0 0) 0x7f6e4415e2a0 con 0x7f6e4415e730 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e30003520 con 0x7f6e4415e730 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000126158 0 0) 0x7f6e30003520 con 0x7f6e4415e730 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 msgr2=0x7f6e4415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e4415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 >> v1:172.21.15.203:6789/0 conn(0x7f6e44143490 legacy=0x7f6e440b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e4415dc50 con 0x7f6e4415e730 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f6e400019f0 con 0x7f6e4415e730 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 >> v1:172.21.15.124:6789/0 conn(0x7f6e4415e730 legacy=0x7f6e4415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e300039a0 0x7f6e30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e4415b520 con 0x7f6e300039a0 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3000ba30 con 0x7f6e300047c0 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3000de00 con 0x7f6e440b1400 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e300039a0 0x7f6e30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e48a56640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49a58640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 -- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 msgr2=0x7f6e30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e440b1400 0x7f6e30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 -- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300047c0 msgr2=0x7f6e30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.511+0000 7f6e49257640 1 -- 172.21.15.124:0/1160450691 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e30003520 con 0x7f6e300039a0 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.512+0000 7f6e48a56640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.512+0000 7f6e49257640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e300039a0 0x7f6e30003dc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c001720 tx=0x7f6e300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc8531934702c2d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.512+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c0052d0 con 0x7f6e300039a0 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.512+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e3c02e050 con 0x7f6e300039a0 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.512+0000 7f6e3bfff640 1 -- 172.21.15.124:0/1160450691 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c0035f0 con 0x7f6e300039a0 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/1160450691 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e300039a0 msgr2=0x7f6e30003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/1160450691 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e300039a0 0x7f6e30003dc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c001720 tx=0x7f6e300057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e49a58640 1 -- 172.21.15.124:0/1160450691 reap_dead start 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/1160450691 shutdown_connections 2023-12-13T23:55:41.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/1160450691 >> 172.21.15.124:0/1160450691 conn(0x7f6e4406a4d0 msgr2=0x7f6e4406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/1160450691 shutdown_connections 2023-12-13T23:55:41.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.513+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/1160450691 wait complete. 2023-12-13T23:55:41.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.514+0000 7f6e4b4bb640 1 Processor -- start 2023-12-13T23:55:41.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.514+0000 7f6e4b4bb640 1 -- start start 2023-12-13T23:55:41.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 0x7f6e441599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e4415c020 0x7f6e4415c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44131910 con 0x7f6e441300e0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e441315c0 con 0x7f6e300039a0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e44130ad0 con 0x7f6e4415c020 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39776/0 (socket says 172.21.15.124:39776) 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 -- 172.21.15.124:0/3155599338 learned_addr learned my addr 172.21.15.124:0/3155599338 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e48a56640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e4415c020 0x7f6e4415c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49257640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 0x7f6e441599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 -- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e4415c020 msgr2=0x7f6e4415c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e4415c020 0x7f6e4415c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 -- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 msgr2=0x7f6e441599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 0x7f6e441599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 -- 172.21.15.124:0/3155599338 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e44159f10 con 0x7f6e441300e0 2023-12-13T23:55:41.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49257640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 0x7f6e441599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e49a58640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6e440cef60 tx=0x7f6e4000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000eb90 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e4000ecf0 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e4415cb30 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.515+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40013660 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.516+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e4415d0f0 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.517+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e440b4380 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.517+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6e400137c0 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.517+0000 7f6e39ffb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 0x7f6e2806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.517+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e40089c80 con 0x7f6e441300e0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.517+0000 7f6e49257640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 0x7f6e2806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.518+0000 7f6e49257640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 0x7f6e2806d6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c033000 tx=0x7f6e3c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.521+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6e40054b00 con 0x7f6e441300e0 2023-12-13T23:55:41.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.688+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e44131a50 con 0x7f6e441300e0 2023-12-13T23:55:41.687 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.688+0000 7f6e39ffb640 1 -- 172.21.15.124:0/3155599338 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e44131a50 con 0x7f6e441300e0 2023-12-13T23:55:41.687 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 msgr2=0x7f6e2806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 0x7f6e2806d6b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c033000 tx=0x7f6e3c009030 comp rx=0 tx=0).stop 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 msgr2=0x7f6e44130590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6e440cef60 tx=0x7f6e4000e530 comp rx=0 tx=0).stop 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 shutdown_connections 2023-12-13T23:55:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6e2806b1f0 0x7f6e2806d6b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6e441300e0 0x7f6e44130590 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6e4415c020 0x7f6e4415c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 --2- 172.21.15.124:0/3155599338 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6e300039a0 0x7f6e441599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 >> 172.21.15.124:0/3155599338 conn(0x7f6e4406a4d0 msgr2=0x7f6e440af640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 shutdown_connections 2023-12-13T23:55:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.690+0000 7f6e4b4bb640 1 -- 172.21.15.124:0/3155599338 wait complete. 2023-12-13T23:55:41.699 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:41.749 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:55:41.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.806+0000 7fcb6d490640 1 Processor -- start 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.806+0000 7fcb6d490640 1 -- start start 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 0x7fcb680b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fcb68160f70 0x7fcb680cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 0x7fcb6806b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb681614e0 con 0x7fcb68160f70 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb681617c0 con 0x7fcb6815d2d0 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb6d490640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb68161aa0 con 0x7fcb680ce480 2023-12-13T23:55:41.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fcb68160f70 0x7fcb680cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fcb68160f70 0x7fcb680cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39782/0 (socket says 172.21.15.124:39782) 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 -- 172.21.15.124:0/1481980793 learned_addr learned my addr 172.21.15.124:0/1481980793 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb677fe640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 0x7fcb6806b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb66ffd640 1 --2- 172.21.15.124:0/1481980793 >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 0x7fcb680b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 -- 172.21.15.124:0/1481980793 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 msgr2=0x7fcb6806b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 0x7fcb6806b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 -- 172.21.15.124:0/1481980793 >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 msgr2=0x7fcb680b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/1481980793 >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 0x7fcb680b8b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb5ffff640 1 -- 172.21.15.124:0/1481980793 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb68161d80 con 0x7fcb68160f70 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb66ffd640 1 --2- 172.21.15.124:0/1481980793 >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 0x7fcb680b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:41.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.807+0000 7fcb677fe640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 0x7fcb6806b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:41.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.808+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb68160f70 0x7fcb680cdf40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcb680b5db0 tx=0x7fcb5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ffc35906e86fc0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.808+0000 7fcb667fc640 1 -- 172.21.15.124:0/1481980793 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50031870 con 0x7fcb68160f70 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.808+0000 7fcb667fc640 1 -- 172.21.15.124:0/1481980793 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb500319d0 con 0x7fcb68160f70 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.808+0000 7fcb667fc640 1 -- 172.21.15.124:0/1481980793 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb50031d60 con 0x7fcb68160f70 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 -- 172.21.15.124:0/1481980793 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb68160f70 msgr2=0x7fcb680cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb68160f70 0x7fcb680cdf40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcb680b5db0 tx=0x7fcb5002b910 comp rx=0 tx=0).stop 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 -- 172.21.15.124:0/1481980793 shutdown_connections 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb68160f70 0x7fcb680cdf40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 --2- 172.21.15.124:0/1481980793 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb680ce480 0x7fcb6806b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 --2- 172.21.15.124:0/1481980793 >> v2:172.21.15.203:3300/0 conn(0x7fcb6815d2d0 0x7fcb680b8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 -- 172.21.15.124:0/1481980793 >> 172.21.15.124:0/1481980793 conn(0x7fcb68072230 msgr2=0x7fcb680cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 -- 172.21.15.124:0/1481980793 shutdown_connections 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.809+0000 7fcb6d490640 1 -- 172.21.15.124:0/1481980793 wait complete. 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.810+0000 7fcb6d490640 1 Processor -- start 2023-12-13T23:55:41.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.810+0000 7fcb6d490640 1 -- start start 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.810+0000 7fcb6d490640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.810+0000 7fcb6d490640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 0x7fcb6815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.810+0000 7fcb6d490640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb68160f70 0x7fcb6815bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb6d490640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680cb4c0 con 0x7fcb680ce480 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb6d490640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680cb7a0 con 0x7fcb6815d2d0 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb6d490640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb680cba80 con 0x7fcb68160f70 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39788/0 (socket says 172.21.15.124:39788) 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 -- 172.21.15.124:0/103044674 learned_addr learned my addr 172.21.15.124:0/103044674 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb677fe640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb68160f70 0x7fcb6815bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 0x7fcb6815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 -- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb68160f70 msgr2=0x7fcb6815bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb68160f70 0x7fcb6815bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 -- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 msgr2=0x7fcb6815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 0x7fcb6815ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 -- 172.21.15.124:0/103044674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb500094d0 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 0x7fcb6815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb66ffd640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c7f0 tx=0x7fcb5800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007a30 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb58007b90 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb680cbd60 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5800ce40 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.811+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68152b60 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.812+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb68069400 con 0x7fcb680ce480 2023-12-13T23:55:41.814 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.813+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fcb580113f0 con 0x7fcb680ce480 2023-12-13T23:55:41.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.813+0000 7fcb5f7fe640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 0x7fcb4806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:41.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.813+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb58089bf0 con 0x7fcb680ce480 2023-12-13T23:55:41.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.813+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 0x7fcb4806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:41.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.814+0000 7fcb5ffff640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 0x7fcb4806d6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb50036000 tx=0x7fcb5003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:41.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.817+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fcb58054920 con 0x7fcb680ce480 2023-12-13T23:55:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.985+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcb680cb600 con 0x7fcb680ce480 2023-12-13T23:55:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.985+0000 7fcb5f7fe640 1 -- 172.21.15.124:0/103044674 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcb5801f020 con 0x7fcb680ce480 2023-12-13T23:55:41.984 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 msgr2=0x7fcb4806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 0x7fcb4806d6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcb50036000 tx=0x7fcb5003c040 comp rx=0 tx=0).stop 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 msgr2=0x7fcb68152620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c7f0 tx=0x7fcb5800cbf0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 shutdown_connections 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcb4806b1f0 0x7fcb4806d6b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fcb6815d2d0 0x7fcb6815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fcb68160f70 0x7fcb6815bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 --2- 172.21.15.124:0/103044674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcb680ce480 0x7fcb68152620 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.987+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 >> 172.21.15.124:0/103044674 conn(0x7fcb68072230 msgr2=0x7fcb680b8910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.988+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 shutdown_connections 2023-12-13T23:55:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:41.988+0000 7fcb6d490640 1 -- 172.21.15.124:0/103044674 wait complete. 2023-12-13T23:55:41.997 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:42.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 Processor -- start 2023-12-13T23:55:42.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 -- start start 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 0x7fe0a415d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40b49f0 con 0x7fe0a40b4e80 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0a40b3de0 con 0x7fe0a415dec0 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.142+0000 7fe0ab083640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a40b1470 con 0x7fe0a40b2e50 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe0a40b4e80 0x7fe0a40b0d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56788/0 (socket says 172.21.15.124:56788) 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a3fff640 1 -- 172.21.15.124:0/1300870679 learned_addr learned my addr 172.21.15.124:0/1300870679 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a9620640 1 --2- 172.21.15.124:0/1300870679 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 0x7fe0a415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339130610 0 0) 0x7fe0a40b49f0 con 0x7fe0a40b4e80 2023-12-13T23:55:42.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe094003520 con 0x7fe0a40b4e80 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224331581 0 0) 0x7fe0a40b3de0 con 0x7fe0a415dec0 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0a40b49f0 con 0x7fe0a415dec0 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a9620640 1 -- 172.21.15.124:0/1300870679 >> v1:172.21.15.203:6789/0 conn(0x7fe0a415dec0 legacy=0x7fe0a4150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2992871313 0 0) 0x7fe094003520 con 0x7fe0a40b4e80 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a9620640 1 -- 172.21.15.124:0/1300870679 >> v1:172.21.15.124:6789/0 conn(0x7fe0a40b4e80 legacy=0x7fe0a40b0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.143+0000 7fe0a9620640 1 -- 172.21.15.124:0/1300870679 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a40b40c0 con 0x7fe0a40b2e50 2023-12-13T23:55:42.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.144+0000 7fe0a9620640 1 --2- 172.21.15.124:0/1300870679 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 0x7fe0a415d200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009dc0 tx=0x7fe09c02bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b35f48e39da921 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.144+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe090001670 con 0x7fe0a40b2e50 2023-12-13T23:55:42.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.144+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c02c880 con 0x7fe0a40b2e50 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.144+0000 7fe0a37fe640 1 -- 172.21.15.124:0/1300870679 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c02cc10 con 0x7fe0a40b2e50 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 -- 172.21.15.124:0/1300870679 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 msgr2=0x7fe0a415d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 --2- 172.21.15.124:0/1300870679 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 0x7fe0a415d200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009dc0 tx=0x7fe09c02bd50 comp rx=0 tx=0).stop 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 -- 172.21.15.124:0/1300870679 shutdown_connections 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 --2- 172.21.15.124:0/1300870679 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b2e50 0x7fe0a415d200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 -- 172.21.15.124:0/1300870679 >> 172.21.15.124:0/1300870679 conn(0x7fe0a406a4d0 msgr2=0x7fe0a406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 -- 172.21.15.124:0/1300870679 shutdown_connections 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.145+0000 7fe0ab083640 1 -- 172.21.15.124:0/1300870679 wait complete. 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.146+0000 7fe0ab083640 1 Processor -- start 2023-12-13T23:55:42.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.146+0000 7fe0ab083640 1 -- start start 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 0x7fe0a41459b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 0x7fe0a4158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415adb0 con 0x7fe0a415dec0 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415b7c0 con 0x7fe0a40b2e50 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0a415bd80 con 0x7fe0a40b4e80 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39790/0 (socket says 172.21.15.124:39790) 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 -- 172.21.15.124:0/2637556826 learned_addr learned my addr 172.21.15.124:0/2637556826 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a3fff640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 0x7fe0a4158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a8e1f640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 0x7fe0a41459b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 -- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 msgr2=0x7fe0a4158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 0x7fe0a4158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 -- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 msgr2=0x7fe0a41459b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 0x7fe0a41459b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 -- 172.21.15.124:0/2637556826 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c009a70 con 0x7fe0a415dec0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a8e1f640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 0x7fe0a41459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a3fff640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 0x7fe0a4158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a9620640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009eb0 tx=0x7fe09c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c034e80 con 0x7fe0a415dec0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c02ce70 con 0x7fe0a415dec0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0a415b000 con 0x7fe0a415dec0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.147+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09c040650 con 0x7fe0a415dec0 2023-12-13T23:55:42.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.148+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0a415ba10 con 0x7fe0a415dec0 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.148+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0a414c040 con 0x7fe0a415dec0 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.149+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe09c0407b0 con 0x7fe0a415dec0 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.149+0000 7fe0a17fa640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 0x7fe08806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.149+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe09c0ac3f0 con 0x7fe0a415dec0 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.149+0000 7fe0a8e1f640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 0x7fe08806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.150 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.150+0000 7fe0a8e1f640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 0x7fe08806d680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0980011e0 tx=0x7fe098009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.154+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe09c077120 con 0x7fe0a415dec0 2023-12-13T23:55:42.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.319+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe0a4069400 con 0x7fe0a415dec0 2023-12-13T23:55:42.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.320+0000 7fe0a17fa640 1 -- 172.21.15.124:0/2637556826 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe09c009450 con 0x7fe0a415dec0 2023-12-13T23:55:42.318 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:42.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.321+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 msgr2=0x7fe08806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.321+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 0x7fe08806d680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe0980011e0 tx=0x7fe098009040 comp rx=0 tx=0).stop 2023-12-13T23:55:42.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.321+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 msgr2=0x7fe0a41589b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.321+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe09c009eb0 tx=0x7fe09c004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:42.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.321+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 shutdown_connections 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe08806b1c0 0x7fe08806d680 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe0a415dec0 0x7fe0a41589b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe0a40b4e80 0x7fe0a4158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 --2- 172.21.15.124:0/2637556826 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe0a40b2e50 0x7fe0a41459b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 >> 172.21.15.124:0/2637556826 conn(0x7fe0a406a4d0 msgr2=0x7fe0a414ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 shutdown_connections 2023-12-13T23:55:42.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.322+0000 7fe0ab083640 1 -- 172.21.15.124:0/2637556826 wait complete. 2023-12-13T23:55:42.364 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2023-12-13T23:55:42.364 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:55:42.365 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:42.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.467+0000 7f0a6c6d4640 1 Processor -- start 2023-12-13T23:55:42.465 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.467+0000 7f0a6c6d4640 1 -- start start 2023-12-13T23:55:42.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6c6d4640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0a64150560 0x7f0a640b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6c6d4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 0x7f0a6415d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6c6d4640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a640b49f0 con 0x7f0a64150940 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6c6d4640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640b3de0 con 0x7f0a64150560 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6c6d4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a640b40c0 con 0x7f0a640b3390 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6a470640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0a64150560 0x7f0a640b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6ac71640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0a64150940 0x7f0a640ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56802/0 (socket says 172.21.15.124:56802) 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6ac71640 1 -- 172.21.15.124:0/2976891678 learned_addr learned my addr 172.21.15.124:0/2976891678 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6a470640 1 -- 172.21.15.124:0/2976891678 >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 msgr2=0x7f0a6415d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a69c6f640 1 --2- 172.21.15.124:0/2976891678 >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 0x7f0a6415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6946e640 1 -- 172.21.15.124:0/2976891678 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726572299 0 0) 0x7f0a640b49f0 con 0x7f0a64150940 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2976891678 >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 0x7f0a6415d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6a470640 1 -- 172.21.15.124:0/2976891678 >> v1:172.21.15.124:6789/0 conn(0x7f0a64150940 legacy=0x7f0a640ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a6a470640 1 -- 172.21.15.124:0/2976891678 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a640b43a0 con 0x7f0a64150560 2023-12-13T23:55:42.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.468+0000 7f0a69c6f640 1 --2- 172.21.15.124:0/2976891678 >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 0x7f0a6415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.469+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2976891678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64150560 0x7f0a640b2e50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a64072810 tx=0x7f0a6002c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b8379d8db90440 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.469+0000 7f0a6946e640 1 -- 172.21.15.124:0/2976891678 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60030800 con 0x7f0a64150560 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.469+0000 7f0a6946e640 1 -- 172.21.15.124:0/2976891678 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60030960 con 0x7f0a64150560 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.469+0000 7f0a6946e640 1 -- 172.21.15.124:0/2976891678 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a60030cf0 con 0x7f0a64150560 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2976891678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64150560 msgr2=0x7f0a640b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2976891678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64150560 0x7f0a640b2e50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a64072810 tx=0x7f0a6002c9e0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2976891678 shutdown_connections 2023-12-13T23:55:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2976891678 >> v2:172.21.15.203:3300/0 conn(0x7f0a640b3390 0x7f0a6415d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2976891678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64150560 0x7f0a640b2e50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2976891678 >> 172.21.15.124:0/2976891678 conn(0x7f0a6406a4d0 msgr2=0x7f0a6406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2976891678 shutdown_connections 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.470+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2976891678 wait complete. 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 Processor -- start 2023-12-13T23:55:42.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 -- start start 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 0x7f0a64144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a64150560 0x7f0a641383a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a64144280 con 0x7f0a64149c40 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a641388e0 con 0x7f0a640b3390 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.471+0000 7f0a6c6d4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a64143fa0 con 0x7f0a64150560 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39804/0 (socket says 172.21.15.124:39804) 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 -- 172.21.15.124:0/2949029412 learned_addr learned my addr 172.21.15.124:0/2949029412 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 0x7f0a64144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a69c6f640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a64150560 0x7f0a641383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 -- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a64150560 msgr2=0x7f0a641383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a64150560 0x7f0a641383a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 -- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 msgr2=0x7f0a64144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 0x7f0a64144a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 -- 172.21.15.124:0/2949029412 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a60009400 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 0x7f0a64144a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6ac71640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c780 tx=0x7f0a5c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00ce90 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c00a640 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00a9d0 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6415a1f0 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.472+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a64138cc0 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.473+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a640b4e60 con 0x7f0a64149c40 2023-12-13T23:55:42.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.474+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0a5c00ab30 con 0x7f0a64149c40 2023-12-13T23:55:42.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.474+0000 7f0a5affd640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 0x7f0a4806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.474+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a5c088eb0 con 0x7f0a64149c40 2023-12-13T23:55:42.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.474+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 0x7f0a4806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.475+0000 7f0a6a470640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 0x7f0a4806d680 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a60037000 tx=0x7f0a6003a040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.478+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0a640b4e60 con 0x7f0a64149c40 2023-12-13T23:55:42.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.650+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a6415a710 con 0x7f0a64149c40 2023-12-13T23:55:42.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.652+0000 7f0a5affd640 1 -- 172.21.15.124:0/2949029412 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f0a6415a710 con 0x7f0a64149c40 2023-12-13T23:55:42.650 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:42.650 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.653+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 msgr2=0x7f0a4806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.653+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 0x7f0a4806d680 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0a60037000 tx=0x7f0a6003a040 comp rx=0 tx=0).stop 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.653+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 msgr2=0x7f0a641432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.653+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00c780 tx=0x7f0a5c00cc40 comp rx=0 tx=0).stop 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 shutdown_connections 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4806b1c0 0x7f0a4806d680 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a64149c40 0x7f0a641432f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a64150560 0x7f0a641383a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 --2- 172.21.15.124:0/2949029412 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a640b3390 0x7f0a64144a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 >> 172.21.15.124:0/2949029412 conn(0x7f0a6406a4d0 msgr2=0x7f0a640b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 shutdown_connections 2023-12-13T23:55:42.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.654+0000 7f0a6c6d4640 1 -- 172.21.15.124:0/2949029412 wait complete. 2023-12-13T23:55:42.663 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:42.663 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:42.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.769+0000 7fa9fa410640 1 Processor -- start 2023-12-13T23:55:42.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.769+0000 7fa9fa410640 1 -- start start 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9f4144980 0x7fa9f415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9f40b4370 0x7fa9f40b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f40b4d20 con 0x7fa9f4144980 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9f4071770 con 0x7fa9f40b4370 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9fa410640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40ccdc0 con 0x7fa9f40ceb70 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9f4144980 0x7fa9f415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9f4144980 0x7fa9f415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39818/0 (socket says 172.21.15.124:39818) 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 -- 172.21.15.124:0/2598710785 learned_addr learned my addr 172.21.15.124:0/2598710785 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:42.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f89ad640 1 --2- 172.21.15.124:0/2598710785 >> v2:172.21.15.203:3300/0 conn(0x7fa9f40b4370 0x7fa9f40b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f37fe640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 -- 172.21.15.124:0/2598710785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 msgr2=0x7fa9f40cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 -- 172.21.15.124:0/2598710785 >> v2:172.21.15.203:3300/0 conn(0x7fa9f40b4370 msgr2=0x7fa9f40b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2598710785 >> v2:172.21.15.203:3300/0 conn(0x7fa9f40b4370 0x7fa9f40b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.770+0000 7fa9f3fff640 1 -- 172.21.15.124:0/2598710785 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f40cd0a0 con 0x7fa9f4144980 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f415c7c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40b2110 tx=0x7fa9e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d18c60016859bd server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9f2ffd640 1 -- 172.21.15.124:0/2598710785 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8031870 con 0x7fa9f4144980 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9f2ffd640 1 -- 172.21.15.124:0/2598710785 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e80319d0 con 0x7fa9f4144980 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9f2ffd640 1 -- 172.21.15.124:0/2598710785 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8031d60 con 0x7fa9f4144980 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 -- 172.21.15.124:0/2598710785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 msgr2=0x7fa9f415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f415c7c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa9f40b2110 tx=0x7fa9e802b910 comp rx=0 tx=0).stop 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 -- 172.21.15.124:0/2598710785 shutdown_connections 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2598710785 >> v2:172.21.15.203:3300/0 conn(0x7fa9f40b4370 0x7fa9f40b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2598710785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f415c7c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.771+0000 7fa9fa410640 1 -- 172.21.15.124:0/2598710785 >> 172.21.15.124:0/2598710785 conn(0x7fa9f406a4d0 msgr2=0x7fa9f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.772+0000 7fa9fa410640 1 -- 172.21.15.124:0/2598710785 shutdown_connections 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.772+0000 7fa9fa410640 1 -- 172.21.15.124:0/2598710785 wait complete. 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.773+0000 7fa9fa410640 1 Processor -- start 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.773+0000 7fa9fa410640 1 -- start start 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f40b4370 0x7fa9f4155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40ae310 con 0x7fa9f4144980 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40b0b50 con 0x7fa9f40b4370 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9f40b0e30 con 0x7fa9f40ceb70 2023-12-13T23:55:42.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39830/0 (socket says 172.21.15.124:39830) 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 -- 172.21.15.124:0/2208233780 learned_addr learned my addr 172.21.15.124:0/2208233780 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f40b4370 0x7fa9f4155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f37fe640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 -- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 msgr2=0x7fa9f40ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 -- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f40b4370 msgr2=0x7fa9f4155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f40b4370 0x7fa9f4155510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 -- 172.21.15.124:0/2208233780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e80094d0 con 0x7fa9f4144980 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f89ad640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00efc0 tx=0x7fa9ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec014480 con 0x7fa9f4144980 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9ec00f040 con 0x7fa9f4144980 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9f40b1110 con 0x7fa9f4144980 2023-12-13T23:55:42.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec014820 con 0x7fa9f4144980 2023-12-13T23:55:42.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.774+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9f40afdb0 con 0x7fa9f4144980 2023-12-13T23:55:42.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.775+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9f415cc80 con 0x7fa9f4144980 2023-12-13T23:55:42.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.776+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa9ec014980 con 0x7fa9f4144980 2023-12-13T23:55:42.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.776+0000 7fa9f0ff9640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 0x7fa9e006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:42.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.776+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9ec08a810 con 0x7fa9f4144980 2023-12-13T23:55:42.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.776+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 0x7fa9e006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:42.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.777+0000 7fa9f3fff640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 0x7fa9e006d680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8036000 tx=0x7fa9e803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:42.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.780+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa9ec0555f0 con 0x7fa9f4144980 2023-12-13T23:55:42.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.947+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9f4069400 con 0x7fa9f4144980 2023-12-13T23:55:42.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.948+0000 7fa9f0ff9640 1 -- 172.21.15.124:0/2208233780 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9ec0180a0 con 0x7fa9f4144980 2023-12-13T23:55:42.946 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 msgr2=0x7fa9e006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 0x7fa9e006d680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8036000 tx=0x7fa9e803c040 comp rx=0 tx=0).stop 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 msgr2=0x7fa9f40addd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00efc0 tx=0x7fa9ec00c490 comp rx=0 tx=0).stop 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 shutdown_connections 2023-12-13T23:55:42.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa9e006b1c0 0x7fa9e006d680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9f4144980 0x7fa9f40addd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9f40ceb70 0x7fa9f40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 --2- 172.21.15.124:0/2208233780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9f40b4370 0x7fa9f4155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.949+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 >> 172.21.15.124:0/2208233780 conn(0x7fa9f406a4d0 msgr2=0x7fa9f4159990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.950+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 shutdown_connections 2023-12-13T23:55:42.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:42.950+0000 7fa9fa410640 1 -- 172.21.15.124:0/2208233780 wait complete. 2023-12-13T23:55:42.959 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:43.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.063+0000 7f2881f73640 1 Processor -- start 2023-12-13T23:55:43.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 -- start start 2023-12-13T23:55:43.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f287c0cebb0 0x7f287c0c5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287c0c7ee0 con 0x7f287c15bbe0 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f287c0cf330 con 0x7f287c0cebb0 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.064+0000 7f2881f73640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c0c7920 con 0x7f287c0c63c0 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f287bfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f287c0cebb0 0x7f287c0c5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f287c0cebb0 0x7f287c0c5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:58664/0 (socket says 172.21.15.124:58664) 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f287bfff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:46390/0 (socket says 172.21.15.124:46390) 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 -- 172.21.15.124:0/1632612709 learned_addr learned my addr 172.21.15.124:0/1632612709 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f287b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f287c15bbe0 0x7f287c15bfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33540/0 (socket says 172.21.15.124:33540) 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 -- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 msgr2=0x7f287c15f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 -- 172.21.15.124:0/1632612709 >> v1:172.21.15.203:6789/0 conn(0x7f287c15bbe0 legacy=0x7f287c15bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f2872ffd640 1 -- 172.21.15.124:0/1632612709 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c0c7c00 con 0x7f287c0cebb0 2023-12-13T23:55:43.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f287affd640 1 -- 172.21.15.124:0/1632612709 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406883501 0 0) 0x7f287c0c7ee0 con 0x7f287c15bbe0 2023-12-13T23:55:43.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.065+0000 7f287bfff640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2872ffd640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0cebb0 0x7f287c0c5e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2864009870 tx=0x7f286402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a24a9c7917ae91b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f287affd640 1 -- 172.21.15.124:0/1632612709 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864031640 con 0x7f287c0cebb0 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f287affd640 1 -- 172.21.15.124:0/1632612709 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28640317a0 con 0x7f287c0cebb0 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f287affd640 1 -- 172.21.15.124:0/1632612709 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2864031b30 con 0x7f287c0cebb0 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 -- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0cebb0 msgr2=0x7f287c0c5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0cebb0 0x7f287c0c5e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2864009870 tx=0x7f286402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 -- 172.21.15.124:0/1632612709 shutdown_connections 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c0c63c0 0x7f287c15f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 --2- 172.21.15.124:0/1632612709 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0cebb0 0x7f287c0c5e80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 -- 172.21.15.124:0/1632612709 >> 172.21.15.124:0/1632612709 conn(0x7f287c0b2c10 msgr2=0x7f287c0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.066+0000 7f2881f73640 1 -- 172.21.15.124:0/1632612709 shutdown_connections 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.067+0000 7f2881f73640 1 -- 172.21.15.124:0/1632612709 wait complete. 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 Processor -- start 2023-12-13T23:55:43.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 -- start start 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0c63c0 0x7f287c14ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c15bbe0 0x7f287c15a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c1e4d60 con 0x7f287c0cebb0 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f287c1e5040 con 0x7f287c0c63c0 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.068+0000 7f2881f73640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f287c1e5320 con 0x7f287c15bbe0 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39846/0 (socket says 172.21.15.124:39846) 2023-12-13T23:55:43.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 -- 172.21.15.124:0/1534191890 learned_addr learned my addr 172.21.15.124:0/1534191890 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f287b7fe640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0c63c0 0x7f287c14ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f287bfff640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c15bbe0 0x7f287c15a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 -- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c15bbe0 msgr2=0x7f287c15a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c15bbe0 0x7f287c15a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 -- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0c63c0 msgr2=0x7f287c14ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0c63c0 0x7f287c14ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 -- 172.21.15.124:0/1534191890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2874009a70 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2872ffd640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2864000c00 tx=0x7f28640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286403a930 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f286403aab0 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28640094d0 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f286403ae60 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.069+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287c1e58e0 con 0x7f287c0cebb0 2023-12-13T23:55:43.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.070+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f287c1e5ad0 con 0x7f287c0cebb0 2023-12-13T23:55:43.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.071+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f286402f050 con 0x7f287c0cebb0 2023-12-13T23:55:43.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.071+0000 7f2878ff9640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 0x7f285c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.071+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f28640ab650 con 0x7f287c0cebb0 2023-12-13T23:55:43.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.071+0000 7f287b7fe640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 0x7f285c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.072+0000 7f287b7fe640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 0x7f285c06d680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f286c007fd0 tx=0x7f286c007de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.075+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2864076410 con 0x7f287c0cebb0 2023-12-13T23:55:43.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.242+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f287c069400 con 0x7f287c0cebb0 2023-12-13T23:55:43.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.243+0000 7f2878ff9640 1 -- 172.21.15.124:0/1534191890 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2864079a30 con 0x7f287c0cebb0 2023-12-13T23:55:43.241 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 msgr2=0x7f285c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 0x7f285c06d680 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f286c007fd0 tx=0x7f286c007de0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 msgr2=0x7f287c1578e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2864000c00 tx=0x7f28640046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 shutdown_connections 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f285c06b1c0 0x7f285c06d680 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f287c15bbe0 0x7f287c15a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f287c0cebb0 0x7f287c1578e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 --2- 172.21.15.124:0/1534191890 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f287c0c63c0 0x7f287c14ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 >> 172.21.15.124:0/1534191890 conn(0x7f287c0b2c10 msgr2=0x7f287c0c71e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:43.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 shutdown_connections 2023-12-13T23:55:43.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.245+0000 7f2881f73640 1 -- 172.21.15.124:0/1534191890 wait complete. 2023-12-13T23:55:43.254 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:43.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.357+0000 7f715d9a5640 1 Processor -- start 2023-12-13T23:55:43.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.357+0000 7f715d9a5640 1 -- start start 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 0x7f71580b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71580d2040 0x7f71580b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71580cded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71580d1bb0 con 0x7f71580d2040 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f715815d6b0 con 0x7f715815d2d0 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f715d9a5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71580ce700 con 0x7f71580cda40 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71580d2040 0x7f71580b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71580d2040 0x7f71580b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39858/0 (socket says 172.21.15.124:39858) 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 -- 172.21.15.124:0/1526236417 learned_addr learned my addr 172.21.15.124:0/1526236417 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 -- 172.21.15.124:0/1526236417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 msgr2=0x7f71580cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 --2- 172.21.15.124:0/1526236417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71580cded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f71567fc640 1 -- 172.21.15.124:0/1526236417 >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 msgr2=0x7f71580b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.358+0000 7f7156ffd640 1 --2- 172.21.15.124:0/1526236417 >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 0x7f71580b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f71567fc640 1 --2- 172.21.15.124:0/1526236417 >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 0x7f71580b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f71567fc640 1 -- 172.21.15.124:0/1526236417 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71580d1280 con 0x7f71580d2040 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f7156ffd640 1 --2- 172.21.15.124:0/1526236417 >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 0x7f71580b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f71567fc640 1 --2- 172.21.15.124:0/1526236417 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f71580b51a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f715806c5d0 tx=0x7f714c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e17c165db0d1fc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f7155ffb640 1 -- 172.21.15.124:0/1526236417 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c031870 con 0x7f71580d2040 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f7155ffb640 1 -- 172.21.15.124:0/1526236417 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c0319d0 con 0x7f71580d2040 2023-12-13T23:55:43.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.359+0000 7f7155ffb640 1 -- 172.21.15.124:0/1526236417 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c031d60 con 0x7f71580d2040 2023-12-13T23:55:43.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 -- 172.21.15.124:0/1526236417 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 msgr2=0x7f71580b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 --2- 172.21.15.124:0/1526236417 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f71580b51a0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f715806c5d0 tx=0x7f714c02b910 comp rx=0 tx=0).stop 2023-12-13T23:55:43.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 -- 172.21.15.124:0/1526236417 shutdown_connections 2023-12-13T23:55:43.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 --2- 172.21.15.124:0/1526236417 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f71580b51a0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 --2- 172.21.15.124:0/1526236417 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71580cded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 --2- 172.21.15.124:0/1526236417 >> v2:172.21.15.203:3300/0 conn(0x7f715815d2d0 0x7f71580b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 -- 172.21.15.124:0/1526236417 >> 172.21.15.124:0/1526236417 conn(0x7f7158072050 msgr2=0x7f7158072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 -- 172.21.15.124:0/1526236417 shutdown_connections 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.360+0000 7f715d9a5640 1 -- 172.21.15.124:0/1526236417 wait complete. 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.361+0000 7f715d9a5640 1 Processor -- start 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.361+0000 7f715d9a5640 1 -- start start 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71581485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 0x7f715814b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7158160280 con 0x7f71580d2040 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7158148b10 con 0x7f715815d2d0 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f715d9a5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7158148f20 con 0x7f71580cda40 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f71567fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f71567fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39866/0 (socket says 172.21.15.124:39866) 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f71567fc640 1 -- 172.21.15.124:0/3988892400 learned_addr learned my addr 172.21.15.124:0/3988892400 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f7156ffd640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71581485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.362+0000 7f71577fe640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 0x7f715814b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 -- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 msgr2=0x7f71581485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71581485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 -- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 msgr2=0x7f715814b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 0x7f715814b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 -- 172.21.15.124:0/3988892400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c0094d0 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71577fe640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 0x7f715814b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f71567fc640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f714c02b950 tx=0x7f714c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c031860 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c0443f0 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7158149200 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c0447a0 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.363+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71581497c0 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.364+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7158069400 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.364+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f714c04d450 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.365+0000 7f713ffff640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 0x7f714406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.365+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f714c02d040 con 0x7f71580d2040 2023-12-13T23:55:43.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.365+0000 7f7156ffd640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 0x7f714406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.365+0000 7f7156ffd640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 0x7f714406d680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7148007510 tx=0x7f71480074a0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.369+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f714c0dee90 con 0x7f71580d2040 2023-12-13T23:55:43.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.535+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f71580b5460 con 0x7f71580d2040 2023-12-13T23:55:43.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.535+0000 7f713ffff640 1 -- 172.21.15.124:0/3988892400 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f714c0350a0 con 0x7f71580d2040 2023-12-13T23:55:43.534 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 msgr2=0x7f714406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 0x7f714406d680 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7148007510 tx=0x7f71480074a0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 msgr2=0x7f715814ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f714c02b950 tx=0x7f714c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 shutdown_connections 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f714406b1c0 0x7f714406d680 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f715815d2d0 0x7f715814b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71580d2040 0x7f715814ae80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 --2- 172.21.15.124:0/3988892400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71580cda40 0x7f71581485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 >> 172.21.15.124:0/3988892400 conn(0x7f7158072050 msgr2=0x7f7158072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:43.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 shutdown_connections 2023-12-13T23:55:43.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.537+0000 7f715d9a5640 1 -- 172.21.15.124:0/3988892400 wait complete. 2023-12-13T23:55:43.547 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2023-12-13T23:55:43.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.648+0000 7ff5a67bb640 1 Processor -- start 2023-12-13T23:55:43.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.648+0000 7ff5a67bb640 1 -- start start 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a67bb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff5a0144980 0x7ff5a015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a67bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a67bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a0071770 con 0x7ff5a00ceb70 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a67bb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a00ccb60 con 0x7ff5a0144980 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a67bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a00b4ce0 con 0x7ff5a00b4370 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff5a0144980 0x7ff5a015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a4d58640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff5a00ceb70 0x7ff5a00cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33544/0 (socket says 172.21.15.124:33544) 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a4d58640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:46442/0 (socket says 172.21.15.124:46442) 2023-12-13T23:55:43.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff5a0144980 0x7ff5a015c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:58688/0 (socket says 172.21.15.124:58688) 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a4d58640 1 -- 172.21.15.124:0/1177765641 learned_addr learned my addr 172.21.15.124:0/1177765641 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 -- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 msgr2=0x7ff5a00b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 -- 172.21.15.124:0/1177765641 >> v1:172.21.15.203:6789/0 conn(0x7ff5a00ceb70 legacy=0x7ff5a00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff59ffff640 1 -- 172.21.15.124:0/1177765641 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00ccdc0 con 0x7ff5a0144980 2023-12-13T23:55:43.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.649+0000 7ff5a4d58640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.650+0000 7ff59ffff640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a0144980 0x7ff5a015c7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097a0 tx=0x7ff59002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc90339fcc1da174 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.650+0000 7ff59effd640 1 -- 172.21.15.124:0/1177765641 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590031830 con 0x7ff5a0144980 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.650+0000 7ff59effd640 1 -- 172.21.15.124:0/1177765641 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff590031990 con 0x7ff5a0144980 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.650+0000 7ff59effd640 1 -- 172.21.15.124:0/1177765641 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff590031d20 con 0x7ff5a0144980 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 -- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a0144980 msgr2=0x7ff5a015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a0144980 0x7ff5a015c7c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5900097a0 tx=0x7ff59002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 -- 172.21.15.124:0/1177765641 shutdown_connections 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a00b4370 0x7ff5a00b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/1177765641 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a0144980 0x7ff5a015c7c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 -- 172.21.15.124:0/1177765641 >> 172.21.15.124:0/1177765641 conn(0x7ff5a006a4d0 msgr2=0x7ff5a006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:43.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 -- 172.21.15.124:0/1177765641 shutdown_connections 2023-12-13T23:55:43.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.651+0000 7ff5a67bb640 1 -- 172.21.15.124:0/1177765641 wait complete. 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.652+0000 7ff5a67bb640 1 Processor -- start 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.652+0000 7ff5a67bb640 1 -- start start 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 0x7ff5a015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a0144980 0x7ff5a0158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a01f36f0 con 0x7ff5a00b4370 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a01f2dc0 con 0x7ff5a00ceb70 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a67bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a01f30a0 con 0x7ff5a0144980 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39878/0 (socket says 172.21.15.124:39878) 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 -- 172.21.15.124:0/2419858014 learned_addr learned my addr 172.21.15.124:0/2419858014 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff5a4d58640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a0144980 0x7ff5a0158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59f7fe640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 0x7ff5a015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 -- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a0144980 msgr2=0x7ff5a0158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a0144980 0x7ff5a0158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 -- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 msgr2=0x7ff5a015c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 0x7ff5a015c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 -- 172.21.15.124:0/2419858014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff598009b30 con 0x7ff5a00b4370 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59f7fe640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 0x7ff5a015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.653+0000 7ff59ffff640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff59002bee0 tx=0x7ff5900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.654+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59003a6f0 con 0x7ff5a00b4370 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.654+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff59003a870 con 0x7ff5a00b4370 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.654+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59003ac20 con 0x7ff5a00b4370 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.654+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff590009400 con 0x7ff5a00b4370 2023-12-13T23:55:43.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.654+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a01f3b80 con 0x7ff5a00b4370 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.655+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a0069400 con 0x7ff5a00b4370 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.655+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff590030380 con 0x7ff5a00b4370 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.655+0000 7ff59cff9640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 0x7ff58c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.656+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff5900ab1e0 con 0x7ff5a00b4370 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.656+0000 7ff59f7fe640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 0x7ff58c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:43.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.656+0000 7ff59f7fe640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 0x7ff58c06d6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff594007fd0 tx=0x7ff594007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:43.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.660+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff590076050 con 0x7ff5a00b4370 2023-12-13T23:55:43.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:43.894+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7ff5a006ada0 con 0x7ff5a00b4370 2023-12-13T23:55:44.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.851+0000 7ff59cff9640 1 -- 172.21.15.124:0/2419858014 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v52) v1 ==== 150+0+0 (secure 0 0 0) 0x7ff590079670 con 0x7ff5a00b4370 2023-12-13T23:55:44.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 msgr2=0x7ff58c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 0x7ff58c06d6b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff594007fd0 tx=0x7ff594007d40 comp rx=0 tx=0).stop 2023-12-13T23:55:44.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 msgr2=0x7ff5a013e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff59002bee0 tx=0x7ff5900046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 shutdown_connections 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff58c06b1f0 0x7ff58c06d6b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5a00ceb70 0x7ff5a015c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5a0144980 0x7ff5a0158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 --2- 172.21.15.124:0/2419858014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5a00b4370 0x7ff5a013e800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.853+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 >> 172.21.15.124:0/2419858014 conn(0x7ff5a006a4d0 msgr2=0x7ff5a00cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.854+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 shutdown_connections 2023-12-13T23:55:44.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.854+0000 7ff5a67bb640 1 -- 172.21.15.124:0/2419858014 wait complete. 2023-12-13T23:55:44.864 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-13T23:55:44.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.973+0000 7f0c7f59e640 1 Processor -- start 2023-12-13T23:55:44.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.973+0000 7f0c7f59e640 1 -- start start 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7f59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c800cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7f59e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c800cebf0 con 0x7f0c800cf310 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7f59e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c800cee90 con 0x7f0c800d00f0 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7f59e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800b2990 con 0x7f0c801607d0 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c77fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0c800cf310 0x7f0c800cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56804/0 (socket says 172.21.15.124:56804) 2023-12-13T23:55:44.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c77fff640 1 -- 172.21.15.124:0/2014456262 learned_addr learned my addr 172.21.15.124:0/2014456262 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212110198 0 0) 0x7f0c800cebf0 con 0x7f0c800cf310 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c6c003520 con 0x7f0c800cf310 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7e59c640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c800cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110691120 0 0) 0x7f0c6c003520 con 0x7f0c800cf310 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 msgr2=0x7f0c800cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c800cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 >> v1:172.21.15.203:6789/0 conn(0x7f0c800d00f0 legacy=0x7f0c800b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c800b2c70 con 0x7f0c800cf310 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0c68001670 con 0x7f0c800cf310 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 >> v1:172.21.15.124:6789/0 conn(0x7f0c800cf310 legacy=0x7f0c800cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c6c0039a0 0x7f0c6c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800b2990 con 0x7f0c6c0047c0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.974+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c00ba30 con 0x7f0c6c0039a0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6c00bbf0 con 0x7f0c801607d0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c7e59c640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c6c0039a0 0x7f0c6c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c7ed9d640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 -- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 msgr2=0x7f0c6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c6c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 -- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c6c0039a0 msgr2=0x7f0c6c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:44.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c6c0039a0 0x7f0c6c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c77fff640 1 -- 172.21.15.124:0/2014456262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c6c003520 con 0x7f0c6c0047c0 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.975+0000 7f0c7ed9d640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801607d0 0x7f0c6c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c77fff640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c6c004c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c68001050 tx=0x7f0c6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195ddae2dc23ca4a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c68005ad0 con 0x7f0c6c0047c0 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c68005c50 con 0x7f0c6c0047c0 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7dd9b640 1 -- 172.21.15.124:0/2014456262 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6802c810 con 0x7f0c6c0047c0 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 -- 172.21.15.124:0/2014456262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 msgr2=0x7f0c6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/2014456262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c6c004c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f0c68001050 tx=0x7f0c6c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7ed9d640 1 -- 172.21.15.124:0/2014456262 reap_dead start 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 -- 172.21.15.124:0/2014456262 shutdown_connections 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 -- 172.21.15.124:0/2014456262 >> 172.21.15.124:0/2014456262 conn(0x7f0c800d2fd0 msgr2=0x7f0c800d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 -- 172.21.15.124:0/2014456262 shutdown_connections 2023-12-13T23:55:44.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.976+0000 7f0c7f59e640 1 -- 172.21.15.124:0/2014456262 wait complete. 2023-12-13T23:55:44.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 Processor -- start 2023-12-13T23:55:44.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 -- start start 2023-12-13T23:55:44.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801f61b0 0x7f0c800d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c800d1410 0x7f0c800b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800d18c0 con 0x7f0c6c0047c0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800b5ba0 con 0x7f0c800d1410 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7f59e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c800cc5b0 con 0x7f0c801f61b0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39890/0 (socket says 172.21.15.124:39890) 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.978+0000 7f0c7e59c640 1 -- 172.21.15.124:0/1255534330 learned_addr learned my addr 172.21.15.124:0/1255534330 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7ed9d640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c800d1410 0x7f0c800b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c77fff640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801f61b0 0x7f0c800d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 -- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801f61b0 msgr2=0x7f0c800d0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801f61b0 0x7f0c800d0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 -- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c800d1410 msgr2=0x7f0c800b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c800d1410 0x7f0c800b6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 -- 172.21.15.124:0/1255534330 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c800b4060 con 0x7f0c6c0047c0 2023-12-13T23:55:44.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7e59c640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0c700119d0 tx=0x7f0c70011e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7000fb50 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7000fcb0 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c801f6640 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c70014440 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.979+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c801f6c00 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.980+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c80069320 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.980+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0c700145a0 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.981+0000 7f0c76ffd640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 0x7f0c6006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.981+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c7008a250 con 0x7f0c6c0047c0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.981+0000 7f0c77fff640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 0x7f0c6006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.982+0000 7f0c77fff640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 0x7f0c6006d6b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0c68034000 tx=0x7f0c68007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:44.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:44.985+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0c70055010 con 0x7f0c6c0047c0 2023-12-13T23:55:45.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.153+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f0c8006ce00 con 0x7f0c6c0047c0 2023-12-13T23:55:45.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.156+0000 7f0c76ffd640 1 -- 172.21.15.124:0/1255534330 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f0c70058630 con 0x7f0c6c0047c0 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 msgr2=0x7f0c6006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 0x7f0c6006d6b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0c68034000 tx=0x7f0c68007000 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 msgr2=0x7f0c800b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0c700119d0 tx=0x7f0c70011e90 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 shutdown_connections 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0c6006b1f0 0x7f0c6006d6b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0c800d1410 0x7f0c800b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c801f61b0 0x7f0c800d0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 --2- 172.21.15.124:0/1255534330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c6c0047c0 0x7f0c800b3b20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 >> 172.21.15.124:0/1255534330 conn(0x7f0c800d2fd0 msgr2=0x7f0c8015e750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:45.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 shutdown_connections 2023-12-13T23:55:45.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.159+0000 7f0c7f59e640 1 -- 172.21.15.124:0/1255534330 wait complete. 2023-12-13T23:55:45.168 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-13T23:55:45.169 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:55:45.207 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:55:45.254 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-13T23:55:45.254 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:45.254 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:45.269 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:45.269 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:45.324 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:45.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.465+0000 7f392ae56640 1 Processor -- start 2023-12-13T23:55:45.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.465+0000 7f392ae56640 1 -- start start 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f392ae56640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f39240b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f392ae56640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f39240d2d90 0x7f39240b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f392ae56640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39240b2b10 con 0x7f39240d2910 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f392ae56640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39240d1590 con 0x7f39240d2d90 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f392ae56640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39240d1870 con 0x7f392415d330 2023-12-13T23:55:45.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f3923fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f39240d2910 0x7f39240d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56812/0 (socket says 172.21.15.124:56812) 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.466+0000 7f3923fff640 1 -- 172.21.15.124:0/4089715992 learned_addr learned my addr 172.21.15.124:0/4089715992 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f39240b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39293f3640 1 --2- 172.21.15.124:0/4089715992 >> v2:172.21.15.203:3300/0 conn(0x7f39240d2d90 0x7f39240b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234586421 0 0) 0x7f39240b2b10 con 0x7f39240d2910 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390c003520 con 0x7f39240d2910 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191268664 0 0) 0x7f390c003520 con 0x7f39240d2910 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 msgr2=0x7f39240b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f39240b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 >> v2:172.21.15.203:3300/0 conn(0x7f39240d2d90 msgr2=0x7f39240b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 --2- 172.21.15.124:0/4089715992 >> v2:172.21.15.203:3300/0 conn(0x7f39240d2d90 0x7f39240b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39240d1b50 con 0x7f39240d2910 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f3918001670 con 0x7f39240d2910 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f39240b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:45.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 >> v1:172.21.15.124:6789/0 conn(0x7f39240d2910 legacy=0x7f39240d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f390c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f390c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f390c004d00 0x7f390c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240d1590 con 0x7f390c003970 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39240d1870 con 0x7f390c004d00 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390c00b980 con 0x7f392415d330 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39293f3640 1 --2- 172.21.15.124:0/4089715992 >> v2:172.21.15.203:3300/0 conn(0x7f39240d2d90 0x7f39240b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f390c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3923fff640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f390c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39293f3640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f390c004d00 0x7f390c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 -- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 msgr2=0x7f390c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f392415d330 0x7f390c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 -- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f390c004d00 msgr2=0x7f390c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f390c004d00 0x7f390c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f3928bf2640 1 -- 172.21.15.124:0/4089715992 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c009a70 con 0x7f390c003970 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.467+0000 7f39293f3640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f390c004d00 0x7f390c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.468+0000 7f3928bf2640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f390c003d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e350 tx=0x7f390c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b827f691aede9ff server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.468+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c02cd60 con 0x7f390c003970 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.468+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c00f040 con 0x7f390c003970 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.468+0000 7f39237fe640 1 -- 172.21.15.124:0/4089715992 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c034640 con 0x7f390c003970 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 -- 172.21.15.124:0/4089715992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 msgr2=0x7f390c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 --2- 172.21.15.124:0/4089715992 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f390c003d90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e350 tx=0x7f390c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f39293f3640 1 -- 172.21.15.124:0/4089715992 reap_dead start 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 -- 172.21.15.124:0/4089715992 shutdown_connections 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 -- 172.21.15.124:0/4089715992 >> 172.21.15.124:0/4089715992 conn(0x7f3924072020 msgr2=0x7f3924072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 -- 172.21.15.124:0/4089715992 shutdown_connections 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.469+0000 7f392ae56640 1 -- 172.21.15.124:0/4089715992 wait complete. 2023-12-13T23:55:45.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.470+0000 7f392ae56640 1 Processor -- start 2023-12-13T23:55:45.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.470+0000 7f392ae56640 1 -- start start 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3924159a10 0x7f39240c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39240c6ba0 0x7f39241f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241f3f50 con 0x7f390c003970 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241f3340 con 0x7f3924159a10 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39241f3620 con 0x7f39240c6ba0 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39922/0 (socket says 172.21.15.124:39922) 2023-12-13T23:55:45.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 -- 172.21.15.124:0/1190831150 learned_addr learned my addr 172.21.15.124:0/1190831150 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 -- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39240c6ba0 msgr2=0x7f39241f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39240c6ba0 0x7f39241f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 -- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3924159a10 msgr2=0x7f39240c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3924159a10 0x7f39240c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 -- 172.21.15.124:0/1190831150 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39241f3900 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f3928bf2640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f391c02c760 tx=0x7f391c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c03e400 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c03e5a0 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c03e950 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39241f3be0 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.471+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39240c7cc0 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.472+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3924069400 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.473+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f391c03eab0 con 0x7f390c003970 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.473+0000 7f39217fa640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 0x7f391006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.473+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f391c0ac010 con 0x7f390c003970 2023-12-13T23:55:45.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.474+0000 7f3923fff640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 0x7f391006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.474+0000 7f3923fff640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 0x7f391006d6b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f39240b45d0 tx=0x7f3918000bc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.477+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f391c07a7e0 con 0x7f390c003970 2023-12-13T23:55:45.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.645+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39240c7cc0 con 0x7f390c003970 2023-12-13T23:55:45.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.646+0000 7f39217fa640 1 -- 172.21.15.124:0/1190831150 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f39240c7cc0 con 0x7f390c003970 2023-12-13T23:55:45.645 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:45.645 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:45.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 msgr2=0x7f391006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 0x7f391006d6b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f39240b45d0 tx=0x7f3918000bc0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 msgr2=0x7f39240c7e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f391c02c760 tx=0x7f391c009750 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 shutdown_connections 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f391006b1f0 0x7f391006d6b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3924159a10 0x7f39240c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f39240c6ba0 0x7f39241f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 --2- 172.21.15.124:0/1190831150 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f390c003970 0x7f39240c7e80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 >> 172.21.15.124:0/1190831150 conn(0x7f3924072020 msgr2=0x7f39240d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 shutdown_connections 2023-12-13T23:55:45.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.648+0000 7f392ae56640 1 -- 172.21.15.124:0/1190831150 wait complete. 2023-12-13T23:55:45.662 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:45.662 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:45.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.766+0000 7f9caed9e640 1 Processor -- start 2023-12-13T23:55:45.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 -- start start 2023-12-13T23:55:45.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9ca80cdb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9ca80cf310 0x7f9ca80b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca80cebf0 con 0x7f9ca81607d0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca80cee90 con 0x7f9ca80cf310 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9caed9e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80b2990 con 0x7f9ca80d00f0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9cadd9c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9ca81607d0 0x7f9ca80cf9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56824/0 (socket says 172.21.15.124:56824) 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.767+0000 7f9cadd9c640 1 -- 172.21.15.124:0/703117057 learned_addr learned my addr 172.21.15.124:0/703117057 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332105886 0 0) 0x7f9ca80cebf0 con 0x7f9ca81607d0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003520 con 0x7f9ca81607d0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 --2- 172.21.15.124:0/703117057 >> v2:172.21.15.203:3300/0 conn(0x7f9ca80cf310 0x7f9ca80b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cad59b640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9ca80cdb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2374724957 0 0) 0x7f9c94003520 con 0x7f9ca81607d0 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 msgr2=0x7f9ca80cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9ca80cdb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 >> v2:172.21.15.203:3300/0 conn(0x7f9ca80cf310 msgr2=0x7f9ca80b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 --2- 172.21.15.124:0/703117057 >> v2:172.21.15.203:3300/0 conn(0x7f9ca80cf310 0x7f9ca80b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80b2c70 con 0x7f9ca81607d0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f9ca4001670 con 0x7f9ca81607d0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 >> v1:172.21.15.124:6789/0 conn(0x7f9ca81607d0 legacy=0x7f9ca80cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9c94003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c94003ef0 0x7f9c940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cee90 con 0x7f9c94004d00 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80b2990 con 0x7f9c94003ef0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9400b980 con 0x7f9ca80d00f0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cadd9c640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9c94003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cad59b640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c94003ef0 0x7f9c940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 -- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 msgr2=0x7f9c94003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9c94003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 -- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c94003ef0 msgr2=0x7f9c940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9c94003ef0 0x7f9c940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cae59d640 1 -- 172.21.15.124:0/703117057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94003520 con 0x7f9c94004d00 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.768+0000 7f9cadd9c640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80d00f0 0x7f9c94003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.769+0000 7f9cae59d640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009fc0 tx=0x7f9c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1b8620359ad929 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.769+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca002cc40 con 0x7f9c94004d00 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.769+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca000f040 con 0x7f9c94004d00 2023-12-13T23:55:45.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.769+0000 7f9cacd9a640 1 -- 172.21.15.124:0/703117057 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0034440 con 0x7f9c94004d00 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 -- 172.21.15.124:0/703117057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c94004d00 msgr2=0x7f9c94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 --2- 172.21.15.124:0/703117057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009fc0 tx=0x7f9c940057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9cae59d640 1 -- 172.21.15.124:0/703117057 reap_dead start 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 -- 172.21.15.124:0/703117057 shutdown_connections 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 -- 172.21.15.124:0/703117057 >> 172.21.15.124:0/703117057 conn(0x7f9ca80d2fd0 msgr2=0x7f9ca80d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 -- 172.21.15.124:0/703117057 shutdown_connections 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.770+0000 7f9caed9e640 1 -- 172.21.15.124:0/703117057 wait complete. 2023-12-13T23:55:45.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.771+0000 7f9caed9e640 1 Processor -- start 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.771+0000 7f9caed9e640 1 -- start start 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ca814f6e0 0x7f9ca80b7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80ccdc0 0x7f9ca80b80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cbf10 con 0x7f9ca814f300 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca81f72b0 con 0x7f9ca814f6e0 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9caed9e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca80cb290 con 0x7f9ca80ccdc0 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39944/0 (socket says 172.21.15.124:39944) 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 -- 172.21.15.124:0/4046069442 learned_addr learned my addr 172.21.15.124:0/4046069442 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cad59b640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ca814f6e0 0x7f9ca80b7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cae59d640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80ccdc0 0x7f9ca80b80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 -- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80ccdc0 msgr2=0x7f9ca80b80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80ccdc0 0x7f9ca80b80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 -- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ca814f6e0 msgr2=0x7f9ca80b7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ca814f6e0 0x7f9ca80b7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 -- 172.21.15.124:0/4046069442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80cb570 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.772+0000 7f9cadd9c640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400eaf0 tx=0x7f9ca400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4004ab0 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca4004c10 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca80cb850 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca400c780 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca8151ce0 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.773+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca80cc050 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.774+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9ca400c8e0 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.774+0000 7f9c9e7fc640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 0x7f9c8c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.774+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ca40891d0 con 0x7f9ca814f300 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.774+0000 7f9cad59b640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 0x7f9c8c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:45.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.775+0000 7f9cad59b640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 0x7f9c8c06d6b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ca81f62f0 tx=0x7f9c98009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:45.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.778+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9ca80cc050 con 0x7f9ca814f300 2023-12-13T23:55:45.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.946+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ca80cc050 con 0x7f9ca814f300 2023-12-13T23:55:45.944 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.946+0000 7f9c9e7fc640 1 -- 172.21.15.124:0/4046069442 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ca80cc050 con 0x7f9ca814f300 2023-12-13T23:55:45.945 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 msgr2=0x7f9c8c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 0x7f9c8c06d6b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9ca81f62f0 tx=0x7f9c98009190 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 msgr2=0x7f9ca81f6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400eaf0 tx=0x7f9ca400eef0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 shutdown_connections 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9c8c06b1f0 0x7f9c8c06d6b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9ca814f6e0 0x7f9ca80b7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9ca80ccdc0 0x7f9ca80b80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 --2- 172.21.15.124:0/4046069442 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9ca814f300 0x7f9ca81f6d70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 >> 172.21.15.124:0/4046069442 conn(0x7f9ca80d2fd0 msgr2=0x7f9ca806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 shutdown_connections 2023-12-13T23:55:45.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:45.948+0000 7f9caed9e640 1 -- 172.21.15.124:0/4046069442 wait complete. 2023-12-13T23:55:45.957 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:46.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.062+0000 7fe8fb66b640 1 Processor -- start 2023-12-13T23:55:46.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.062+0000 7fe8fb66b640 1 -- start start 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 0x7fe8f40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8f415a4e0 0x7fe8f415a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f415b520 con 0x7fe8f415a4e0 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8f4143870 con 0x7fe8f415e730 2023-12-13T23:55:46.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8fb66b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f415e2a0 con 0x7fe8f4143490 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8f415a4e0 0x7fe8f415a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8f415a4e0 0x7fe8f415a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39950/0 (socket says 172.21.15.124:39950) 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 -- 172.21.15.124:0/3080561824 learned_addr learned my addr 172.21.15.124:0/3080561824 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9407640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f8c06640 1 --2- 172.21.15.124:0/3080561824 >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 0x7fe8f40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 -- 172.21.15.124:0/3080561824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 msgr2=0x7fe8f40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 -- 172.21.15.124:0/3080561824 >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 msgr2=0x7fe8f40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 --2- 172.21.15.124:0/3080561824 >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 0x7fe8f40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9c08640 1 -- 172.21.15.124:0/3080561824 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f415b240 con 0x7fe8f415a4e0 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f9407640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.063+0000 7fe8f8c06640 1 --2- 172.21.15.124:0/3080561824 >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 0x7fe8f40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:46.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8f9c08640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f415a970 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8f406e680 tx=0x7fe8ec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e13107d0af2face5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8e7fff640 1 -- 172.21.15.124:0/3080561824 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec02ca70 con 0x7fe8f415a4e0 2023-12-13T23:55:46.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8e7fff640 1 -- 172.21.15.124:0/3080561824 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec02cbd0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8e7fff640 1 -- 172.21.15.124:0/3080561824 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec0343f0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 -- 172.21.15.124:0/3080561824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 msgr2=0x7fe8f415a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f415a970 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8f406e680 tx=0x7fe8ec02bed0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 -- 172.21.15.124:0/3080561824 shutdown_connections 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f415a970 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/3080561824 >> v2:172.21.15.203:3300/0 conn(0x7fe8f415e730 0x7fe8f40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/3080561824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f40b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.064+0000 7fe8fb66b640 1 -- 172.21.15.124:0/3080561824 >> 172.21.15.124:0/3080561824 conn(0x7fe8f406a4d0 msgr2=0x7fe8f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.065+0000 7fe8fb66b640 1 -- 172.21.15.124:0/3080561824 shutdown_connections 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.065+0000 7fe8fb66b640 1 -- 172.21.15.124:0/3080561824 wait complete. 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 Processor -- start 2023-12-13T23:55:46.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 -- start start 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f413e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8f415e730 0x7fe8f4141c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f413e860 con 0x7fe8f415a4e0 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40ab4d0 con 0x7fe8f415e730 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8fb66b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8f40ab1f0 con 0x7fe8f4143490 2023-12-13T23:55:46.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8f8c06640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8f8c06640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39958/0 (socket says 172.21.15.124:39958) 2023-12-13T23:55:46.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.066+0000 7fe8f8c06640 1 -- 172.21.15.124:0/592709758 learned_addr learned my addr 172.21.15.124:0/592709758 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:46.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f9c08640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8f415e730 0x7fe8f4141c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f9407640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f413e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 -- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 msgr2=0x7fe8f413e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f413e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 -- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8f415e730 msgr2=0x7fe8f4141c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8f415e730 0x7fe8f4141c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 -- 172.21.15.124:0/592709758 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec009b30 con 0x7fe8f415a4e0 2023-12-13T23:55:46.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8f8c06640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c7f0 tx=0x7fe8f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0007820 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f0007980 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8f40abeb0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f0007cf0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.067+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8f40ac470 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.068+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8f40b4380 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.069+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe8f00113f0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.069+0000 7fe8e5ffb640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 0x7fe8d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.069+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe8f0089c20 con 0x7fe8f415a4e0 2023-12-13T23:55:46.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.069+0000 7fe8f9407640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 0x7fe8d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.070+0000 7fe8f9407640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 0x7fe8d806d680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800a990 tx=0x7fe8e8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.073+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe8f00549e0 con 0x7fe8f415a4e0 2023-12-13T23:55:46.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.239+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe8f4069400 con 0x7fe8f415a4e0 2023-12-13T23:55:46.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.240+0000 7fe8e5ffb640 1 -- 172.21.15.124:0/592709758 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe8f001f020 con 0x7fe8f415a4e0 2023-12-13T23:55:46.239 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:46.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 msgr2=0x7fe8d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 0x7fe8d806d680 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800a990 tx=0x7fe8e8008040 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 msgr2=0x7fe8f4141720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fe8f000c7f0 tx=0x7fe8f000ccb0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 shutdown_connections 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8d806b1c0 0x7fe8d806d680 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8f415e730 0x7fe8f4141c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8f415a4e0 0x7fe8f4141720 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 --2- 172.21.15.124:0/592709758 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8f4143490 0x7fe8f413e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 >> 172.21.15.124:0/592709758 conn(0x7fe8f406a4d0 msgr2=0x7fe8f40b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 shutdown_connections 2023-12-13T23:55:46.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.242+0000 7fe8fb66b640 1 -- 172.21.15.124:0/592709758 wait complete. 2023-12-13T23:55:46.251 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:46.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 Processor -- start 2023-12-13T23:55:46.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 -- start start 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe84415d270 0x7fe84406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8440709c0 0x7fe8440cd7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8440d0c40 con 0x7fe8440cdd10 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440ce1a0 con 0x7fe8440709c0 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.355+0000 7fe84abd7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8440ced90 con 0x7fe84415d270 2023-12-13T23:55:46.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8440709c0 0x7fe8440cd7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe8440709c0 0x7fe8440cd7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39970/0 (socket says 172.21.15.124:39970) 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 -- 172.21.15.124:0/3365290507 learned_addr learned my addr 172.21.15.124:0/3365290507 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe848973640 1 --2- 172.21.15.124:0/3365290507 >> v2:172.21.15.203:3300/0 conn(0x7fe84415d270 0x7fe84406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 -- 172.21.15.124:0/3365290507 >> v2:172.21.15.203:3300/0 conn(0x7fe84415d270 msgr2=0x7fe84406ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 --2- 172.21.15.124:0/3365290507 >> v2:172.21.15.203:3300/0 conn(0x7fe84415d270 0x7fe84406ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 -- 172.21.15.124:0/3365290507 >> v1:172.21.15.124:6789/0 conn(0x7fe8440cdd10 legacy=0x7fe8440cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.356+0000 7fe843fff640 1 -- 172.21.15.124:0/3365290507 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440cf070 con 0x7fe8440709c0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe843fff640 1 --2- 172.21.15.124:0/3365290507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440709c0 0x7fe8440cd7d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8440ceac0 tx=0x7fe83002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d422faa33861464 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe8437fe640 1 -- 172.21.15.124:0/3365290507 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830031870 con 0x7fe8440709c0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe8437fe640 1 -- 172.21.15.124:0/3365290507 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8300319d0 con 0x7fe8440709c0 2023-12-13T23:55:46.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe8437fe640 1 -- 172.21.15.124:0/3365290507 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830031d60 con 0x7fe8440709c0 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 -- 172.21.15.124:0/3365290507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440709c0 msgr2=0x7fe8440cd7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 --2- 172.21.15.124:0/3365290507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440709c0 0x7fe8440cd7d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe8440ceac0 tx=0x7fe83002b910 comp rx=0 tx=0).stop 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 -- 172.21.15.124:0/3365290507 shutdown_connections 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 --2- 172.21.15.124:0/3365290507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440709c0 0x7fe8440cd7d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 --2- 172.21.15.124:0/3365290507 >> v2:172.21.15.203:3300/0 conn(0x7fe84415d270 0x7fe84406ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 -- 172.21.15.124:0/3365290507 >> 172.21.15.124:0/3365290507 conn(0x7fe8440b2c10 msgr2=0x7fe8440b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 -- 172.21.15.124:0/3365290507 shutdown_connections 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.357+0000 7fe84abd7640 1 -- 172.21.15.124:0/3365290507 wait complete. 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.358+0000 7fe84abd7640 1 Processor -- start 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.358+0000 7fe84abd7640 1 -- start start 2023-12-13T23:55:46.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8440709c0 0x7fe8440caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 0x7fe8440ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8440c9890 con 0x7fe8440cdd10 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84415e5d0 con 0x7fe8440709c0 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe84abd7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84415e8b0 con 0x7fe84415d270 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe843fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe843fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39972/0 (socket says 172.21.15.124:39972) 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.359+0000 7fe843fff640 1 -- 172.21.15.124:0/2669638317 learned_addr learned my addr 172.21.15.124:0/2669638317 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe848973640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8440709c0 0x7fe8440caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe849174640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 0x7fe8440ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 -- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 msgr2=0x7fe8440ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 0x7fe8440ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 -- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8440709c0 msgr2=0x7fe8440caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8440709c0 0x7fe8440caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 -- 172.21.15.124:0/2669638317 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8300094d0 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe849174640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 0x7fe8440ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe843fff640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe830000c00 tx=0x7fe8300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83003abe0 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83003ad40 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84415eb90 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830044610 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.360+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84415bf80 con 0x7fe8440cdd10 2023-12-13T23:55:46.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.361+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844067fc0 con 0x7fe8440cdd10 2023-12-13T23:55:46.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.361+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe83004d450 con 0x7fe8440cdd10 2023-12-13T23:55:46.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.362+0000 7fe8417fa640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 0x7fe82806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.362+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe83002d040 con 0x7fe8440cdd10 2023-12-13T23:55:46.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.362+0000 7fe848973640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 0x7fe82806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.362+0000 7fe848973640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 0x7fe82806d680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe838006fd0 tx=0x7fe838008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.366+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe830076d30 con 0x7fe8440cdd10 2023-12-13T23:55:46.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.535+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe84415f0b0 con 0x7fe8440cdd10 2023-12-13T23:55:46.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.535+0000 7fe8417fa640 1 -- 172.21.15.124:0/2669638317 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe84415f0b0 con 0x7fe8440cdd10 2023-12-13T23:55:46.534 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:46.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 msgr2=0x7fe82806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 0x7fe82806d680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe838006fd0 tx=0x7fe838008040 comp rx=0 tx=0).stop 2023-12-13T23:55:46.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 msgr2=0x7fe84415ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fe830000c00 tx=0x7fe8300046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 shutdown_connections 2023-12-13T23:55:46.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe82806b1c0 0x7fe82806d680 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe84415d270 0x7fe8440ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8440cdd10 0x7fe84415ba40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 --2- 172.21.15.124:0/2669638317 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8440709c0 0x7fe8440caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 >> 172.21.15.124:0/2669638317 conn(0x7fe8440b2c10 msgr2=0x7fe8440b3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 shutdown_connections 2023-12-13T23:55:46.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.537+0000 7fe84abd7640 1 -- 172.21.15.124:0/2669638317 wait complete. 2023-12-13T23:55:46.547 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:46.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.650+0000 7fea4a5e5640 1 Processor -- start 2023-12-13T23:55:46.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.650+0000 7fea4a5e5640 1 -- start start 2023-12-13T23:55:46.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea4a5e5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea440d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea4a5e5640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440b2b10 con 0x7fea4415d330 2023-12-13T23:55:46.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea4a5e5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea440d3b50 con 0x7fea440d3730 2023-12-13T23:55:46.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea4a5e5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea440d1590 con 0x7fea440d2910 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea43fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fea4415d330 0x7fea440b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56832/0 (socket says 172.21.15.124:56832) 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea43fff640 1 -- 172.21.15.124:0/4287849699 learned_addr learned my addr 172.21.15.124:0/4287849699 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea437fe640 1 --2- 172.21.15.124:0/4287849699 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea440d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880227883 0 0) 0x7fea440b2b10 con 0x7fea4415d330 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c003520 con 0x7fea4415d330 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592087902 0 0) 0x7fea440d3b50 con 0x7fea440d3730 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea440b2b10 con 0x7fea440d3730 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea437fe640 1 -- 172.21.15.124:0/4287849699 >> v1:172.21.15.203:6789/0 conn(0x7fea440d3730 legacy=0x7fea440b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea437fe640 1 -- 172.21.15.124:0/4287849699 >> v1:172.21.15.124:6789/0 conn(0x7fea4415d330 legacy=0x7fea440b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.651+0000 7fea437fe640 1 -- 172.21.15.124:0/4287849699 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea440d1870 con 0x7fea440d2910 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.652+0000 7fea437fe640 1 --2- 172.21.15.124:0/4287849699 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea440d31f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea380097d0 tx=0x7fea3802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7030c7d4645d3c40 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.652+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38031800 con 0x7fea440d2910 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea38031960 con 0x7fea440d2910 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea42ffd640 1 -- 172.21.15.124:0/4287849699 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38031cf0 con 0x7fea440d2910 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 -- 172.21.15.124:0/4287849699 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 msgr2=0x7fea440d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/4287849699 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea440d31f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fea380097d0 tx=0x7fea3802b8a0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 -- 172.21.15.124:0/4287849699 shutdown_connections 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/4287849699 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea440d31f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 -- 172.21.15.124:0/4287849699 >> 172.21.15.124:0/4287849699 conn(0x7fea44072020 msgr2=0x7fea44072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 -- 172.21.15.124:0/4287849699 shutdown_connections 2023-12-13T23:55:46.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.653+0000 7fea4a5e5640 1 -- 172.21.15.124:0/4287849699 wait complete. 2023-12-13T23:55:46.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.654+0000 7fea4a5e5640 1 Processor -- start 2023-12-13T23:55:46.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.654+0000 7fea4a5e5640 1 -- start start 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea4415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea4415d330 0x7fea440cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440cbed0 con 0x7fea440d3730 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea440cc1b0 con 0x7fea4415d330 2023-12-13T23:55:46.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea4a5e5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea440cc490 con 0x7fea440d2910 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39978/0 (socket says 172.21.15.124:39978) 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 -- 172.21.15.124:0/2774395684 learned_addr learned my addr 172.21.15.124:0/2774395684 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 -- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 msgr2=0x7fea4415d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea4415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 -- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea4415d330 msgr2=0x7fea440cb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea4415d330 0x7fea440cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.655+0000 7fea437fe640 1 -- 172.21.15.124:0/2774395684 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38009400 con 0x7fea440d3730 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea437fe640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fea38034040 tx=0x7fea380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3803ac60 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3803ae00 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea440cc770 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38044630 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.656+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea4415dec0 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.657+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4415e0b0 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.657+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fea3804d450 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.658+0000 7fea40ff9640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 0x7fea3006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.658+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fea3802e070 con 0x7fea440d3730 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.658+0000 7fea43fff640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 0x7fea3006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:46.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.658+0000 7fea43fff640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 0x7fea3006d680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea34005880 tx=0x7fea34007950 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:46.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.662+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fea38076aa0 con 0x7fea440d3730 2023-12-13T23:55:46.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.842+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fea440ccc90 con 0x7fea440d3730 2023-12-13T23:55:46.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.842+0000 7fea40ff9640 1 -- 172.21.15.124:0/2774395684 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fea440ccc90 con 0x7fea440d3730 2023-12-13T23:55:46.841 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:46.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 msgr2=0x7fea3006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 0x7fea3006d680 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fea34005880 tx=0x7fea34007950 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 msgr2=0x7fea440cb380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fea38034040 tx=0x7fea380046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 shutdown_connections 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fea3006b1c0 0x7fea3006d680 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea4415d330 0x7fea440cb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea440d3730 0x7fea440cb380 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 --2- 172.21.15.124:0/2774395684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea440d2910 0x7fea4415d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 >> 172.21.15.124:0/2774395684 conn(0x7fea44072020 msgr2=0x7fea440d0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 shutdown_connections 2023-12-13T23:55:46.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:46.844+0000 7fea4a5e5640 1 -- 172.21.15.124:0/2774395684 wait complete. 2023-12-13T23:55:46.854 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:46.854 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:46.870 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:47.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.012+0000 7ff37b197640 1 Processor -- start 2023-12-13T23:55:47.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.012+0000 7ff37b197640 1 -- start start 2023-12-13T23:55:47.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 0x7ff374071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff3740ae480 0x7ff3740721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff37406d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740ae910 con 0x7ff3740ae480 2023-12-13T23:55:47.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3740b2200 con 0x7ff37415d2b0 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff37b197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3740b24e0 con 0x7ff374072840 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff3740ae480 0x7ff3740721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff3740ae480 0x7ff3740721c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39986/0 (socket says 172.21.15.124:39986) 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 -- 172.21.15.124:0/1257586190 learned_addr learned my addr 172.21.15.124:0/1257586190 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff379734640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff37406d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 -- 172.21.15.124:0/1257586190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 msgr2=0x7ff37406d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff378f33640 1 --2- 172.21.15.124:0/1257586190 >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 0x7ff374071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff37406d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 -- 172.21.15.124:0/1257586190 >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 msgr2=0x7ff374071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 --2- 172.21.15.124:0/1257586190 >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 0x7ff374071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff373fff640 1 -- 172.21.15.124:0/1257586190 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740b40a0 con 0x7ff3740ae480 2023-12-13T23:55:47.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff379734640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff37406d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.013+0000 7ff378f33640 1 --2- 172.21.15.124:0/1257586190 >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 0x7ff374071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff373fff640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3740ae480 0x7ff3740721c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff374072cc0 tx=0x7ff36002b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1173994f32de1163 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff3737fe640 1 -- 172.21.15.124:0/1257586190 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600317d0 con 0x7ff3740ae480 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff3737fe640 1 -- 172.21.15.124:0/1257586190 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360031930 con 0x7ff3740ae480 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff3737fe640 1 -- 172.21.15.124:0/1257586190 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360031cc0 con 0x7ff3740ae480 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff37b197640 1 -- 172.21.15.124:0/1257586190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3740ae480 msgr2=0x7ff3740721c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.014+0000 7ff37b197640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3740ae480 0x7ff3740721c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff374072cc0 tx=0x7ff36002b870 comp rx=0 tx=0).stop 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 -- 172.21.15.124:0/1257586190 shutdown_connections 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3740ae480 0x7ff3740721c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 --2- 172.21.15.124:0/1257586190 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff37406d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 --2- 172.21.15.124:0/1257586190 >> v2:172.21.15.203:3300/0 conn(0x7ff37415d2b0 0x7ff374071c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 -- 172.21.15.124:0/1257586190 >> 172.21.15.124:0/1257586190 conn(0x7ff3740b4720 msgr2=0x7ff3740b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 -- 172.21.15.124:0/1257586190 shutdown_connections 2023-12-13T23:55:47.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.015+0000 7ff37b197640 1 -- 172.21.15.124:0/1257586190 wait complete. 2023-12-13T23:55:47.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.016+0000 7ff37b197640 1 Processor -- start 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.016+0000 7ff37b197640 1 -- start start 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff374146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff374146cd0 0x7ff374154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff374154980 con 0x7ff37415d2b0 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3741453e0 con 0x7ff374146cd0 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3741456c0 con 0x7ff374072840 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39994/0 (socket says 172.21.15.124:39994) 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 -- 172.21.15.124:0/1561051072 learned_addr learned my addr 172.21.15.124:0/1561051072 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff378f33640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff374146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff379734640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff374146cd0 0x7ff374154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 -- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 msgr2=0x7ff374146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff374146790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 -- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff374146cd0 msgr2=0x7ff374154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff374146cd0 0x7ff374154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 -- 172.21.15.124:0/1561051072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600094d0 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff378f33640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff374146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff373fff640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff3600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3600328f0 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360032a70 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3740b2af0 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360032e20 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.017+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff374145c80 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.018+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3740b2650 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.019+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff360030070 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.019+0000 7ff3717fa640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 0x7ff35806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.019+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff3600ab660 con 0x7ff37415d2b0 2023-12-13T23:55:47.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.019+0000 7ff378f33640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 0x7ff35806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.022 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.020+0000 7ff378f33640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 0x7ff35806d680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff368006fd0 tx=0x7ff368008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.022 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.023+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff360076440 con 0x7ff37415d2b0 2023-12-13T23:55:47.188 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.189+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff374069400 con 0x7ff37415d2b0 2023-12-13T23:55:47.189 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.190+0000 7ff3717fa640 1 -- 172.21.15.124:0/1561051072 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7ff360079a60 con 0x7ff37415d2b0 2023-12-13T23:55:47.189 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:47.190 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 msgr2=0x7ff35806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 0x7ff35806d680 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff368006fd0 tx=0x7ff368008040 comp rx=0 tx=0).stop 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 msgr2=0x7ff374153f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff360000c00 tx=0x7ff3600046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 shutdown_connections 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff35806b1c0 0x7ff35806d680 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.191 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff374146cd0 0x7ff374154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff37415d2b0 0x7ff374153f00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 --2- 172.21.15.124:0/1561051072 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff374072840 0x7ff374146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 >> 172.21.15.124:0/1561051072 conn(0x7ff3740b4720 msgr2=0x7ff374158050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 shutdown_connections 2023-12-13T23:55:47.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.192+0000 7ff37b197640 1 -- 172.21.15.124:0/1561051072 wait complete. 2023-12-13T23:55:47.202 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:47.202 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:47.304 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.306+0000 7fa4c7fd7640 1 Processor -- start 2023-12-13T23:55:47.305 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.306+0000 7fa4c7fd7640 1 -- start start 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c7fd7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa4c0143490 0x7fa4c00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c7fd7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa4c015e730 0x7fa4c00b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c7fd7640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c015e2a0 con 0x7fa4c015a4e0 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c7fd7640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c015b520 con 0x7fa4c015e730 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c7fd7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c0143870 con 0x7fa4c0143490 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa4c015e730 0x7fa4c00b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa4c015e730 0x7fa4c00b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40002/0 (socket says 172.21.15.124:40002) 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 -- 172.21.15.124:0/4070311736 learned_addr learned my addr 172.21.15.124:0/4070311736 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5d73640 1 --2- 172.21.15.124:0/4070311736 >> v2:172.21.15.203:3300/0 conn(0x7fa4c0143490 0x7fa4c00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 -- 172.21.15.124:0/4070311736 >> v2:172.21.15.203:3300/0 conn(0x7fa4c0143490 msgr2=0x7fa4c00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 --2- 172.21.15.124:0/4070311736 >> v2:172.21.15.203:3300/0 conn(0x7fa4c0143490 0x7fa4c00b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 -- 172.21.15.124:0/4070311736 >> v1:172.21.15.124:6789/0 conn(0x7fa4c015a4e0 legacy=0x7fa4c015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.307+0000 7fa4c5572640 1 -- 172.21.15.124:0/4070311736 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c015b240 con 0x7fa4c015e730 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.308+0000 7fa4c5572640 1 --2- 172.21.15.124:0/4070311736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015e730 0x7fa4c00b1230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa4c006e620 tx=0x7fa4b002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853f84de055fabe server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.308+0000 7fa4c4d71640 1 -- 172.21.15.124:0/4070311736 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0030860 con 0x7fa4c015e730 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.308+0000 7fa4c4d71640 1 -- 172.21.15.124:0/4070311736 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b00309c0 con 0x7fa4c015e730 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.308+0000 7fa4c4d71640 1 -- 172.21.15.124:0/4070311736 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0030d50 con 0x7fa4c015e730 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/4070311736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015e730 msgr2=0x7fa4c00b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/4070311736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015e730 0x7fa4c00b1230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa4c006e620 tx=0x7fa4b002b9a0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/4070311736 shutdown_connections 2023-12-13T23:55:47.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/4070311736 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015e730 0x7fa4c00b1230 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/4070311736 >> v2:172.21.15.203:3300/0 conn(0x7fa4c0143490 0x7fa4c00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/4070311736 >> 172.21.15.124:0/4070311736 conn(0x7fa4c006a4d0 msgr2=0x7fa4c006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/4070311736 shutdown_connections 2023-12-13T23:55:47.309 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.309+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/4070311736 wait complete. 2023-12-13T23:55:47.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.310+0000 7fa4c7fd7640 1 Processor -- start 2023-12-13T23:55:47.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.310+0000 7fa4c7fd7640 1 -- start start 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.310+0000 7fa4c7fd7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 0x7fa4c01e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.310+0000 7fa4c7fd7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.310+0000 7fa4c7fd7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 0x7fa4c013ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c7fd7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c013c710 con 0x7fa4c015a4e0 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c7fd7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c01f0100 con 0x7fa4c015e730 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c7fd7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c013d120 con 0x7fa4c0143490 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40008/0 (socket says 172.21.15.124:40008) 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 -- 172.21.15.124:0/1590694651 learned_addr learned my addr 172.21.15.124:0/1590694651 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5d73640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 0x7fa4c01e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c6574640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 0x7fa4c013ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 -- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 msgr2=0x7fa4c01e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 0x7fa4c01e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 -- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 msgr2=0x7fa4c013ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 0x7fa4c013ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 -- 172.21.15.124:0/1590694651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b00094d0 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5d73640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 0x7fa4c01e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c6574640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 0x7fa4c013ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c5572640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0000c00 tx=0x7fa4b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0030830 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b0039d90 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c013cb60 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b0043650 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.311+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c013ef40 con 0x7fa4c015a4e0 2023-12-13T23:55:47.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.312+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c01f0240 con 0x7fa4c015a4e0 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.313+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa4b004c3b0 con 0x7fa4c015a4e0 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.313+0000 7fa4b67fc640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 0x7fa4a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.313+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4b00ac190 con 0x7fa4c015a4e0 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.313+0000 7fa4c5d73640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 0x7fa4a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.314+0000 7fa4c5d73640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 0x7fa4a406d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc008890 tx=0x7fa4bc008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.317+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa4c01f0240 con 0x7fa4c015a4e0 2023-12-13T23:55:47.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.483+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4c01f0240 con 0x7fa4c015a4e0 2023-12-13T23:55:47.482 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.483+0000 7fa4b67fc640 1 -- 172.21.15.124:0/1590694651 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4c01f0240 con 0x7fa4c015a4e0 2023-12-13T23:55:47.482 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 msgr2=0x7fa4a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 0x7fa4a406d680 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc008890 tx=0x7fa4bc008040 comp rx=0 tx=0).stop 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 msgr2=0x7fa4c013c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fa4b0000c00 tx=0x7fa4b0004730 comp rx=0 tx=0).stop 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 shutdown_connections 2023-12-13T23:55:47.484 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a406b1c0 0x7fa4a406d680 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4c015e730 0x7fa4c013ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4c015a4e0 0x7fa4c013c010 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 --2- 172.21.15.124:0/1590694651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4c0143490 0x7fa4c01e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 >> 172.21.15.124:0/1590694651 conn(0x7fa4c006a4d0 msgr2=0x7fa4c00b3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 shutdown_connections 2023-12-13T23:55:47.485 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.485+0000 7fa4c7fd7640 1 -- 172.21.15.124:0/1590694651 wait complete. 2023-12-13T23:55:47.497 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:47.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.602+0000 7ff913790640 1 Processor -- start 2023-12-13T23:55:47.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.602+0000 7ff913790640 1 -- start start 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff90c143490 0x7ff90c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c15e730 0x7ff90c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 0x7ff90c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c15b520 con 0x7ff90c143490 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff90c143870 con 0x7ff90c15a4e0 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff913790640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c15e2a0 con 0x7ff90c15e730 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff91152c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff90c143490 0x7ff90c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff91152c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff90c143490 0x7ff90c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40016/0 (socket says 172.21.15.124:40016) 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.603+0000 7ff91152c640 1 -- 172.21.15.124:0/1738233188 learned_addr learned my addr 172.21.15.124:0/1738233188 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff911d2d640 1 --2- 172.21.15.124:0/1738233188 >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 0x7ff90c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff910d2b640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c15e730 0x7ff90c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 -- 172.21.15.124:0/1738233188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c15e730 msgr2=0x7ff90c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c15e730 0x7ff90c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 -- 172.21.15.124:0/1738233188 >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 msgr2=0x7ff90c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 --2- 172.21.15.124:0/1738233188 >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 0x7ff90c15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 -- 172.21.15.124:0/1738233188 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c15b240 con 0x7ff90c143490 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff911d2d640 1 --2- 172.21.15.124:0/1738233188 >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 0x7ff90c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff91152c640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c143490 0x7ff90c0b0cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff90c15eb90 tx=0x7ff90802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc99a68144a186a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.604+0000 7ff903fff640 1 -- 172.21.15.124:0/1738233188 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90802ea80 con 0x7ff90c143490 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff903fff640 1 -- 172.21.15.124:0/1738233188 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff90802ebe0 con 0x7ff90c143490 2023-12-13T23:55:47.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff903fff640 1 -- 172.21.15.124:0/1738233188 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff908009600 con 0x7ff90c143490 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 -- 172.21.15.124:0/1738233188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c143490 msgr2=0x7ff90c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c143490 0x7ff90c0b0cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff90c15eb90 tx=0x7ff90802e0e0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 -- 172.21.15.124:0/1738233188 shutdown_connections 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 --2- 172.21.15.124:0/1738233188 >> v2:172.21.15.203:3300/0 conn(0x7ff90c15a4e0 0x7ff90c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c15e730 0x7ff90c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 --2- 172.21.15.124:0/1738233188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c143490 0x7ff90c0b0cf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 -- 172.21.15.124:0/1738233188 >> 172.21.15.124:0/1738233188 conn(0x7ff90c06a4d0 msgr2=0x7ff90c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 -- 172.21.15.124:0/1738233188 shutdown_connections 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.605+0000 7ff913790640 1 -- 172.21.15.124:0/1738233188 wait complete. 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.606+0000 7ff913790640 1 Processor -- start 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.606+0000 7ff913790640 1 -- start start 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c143490 0x7ff90c0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 0x7ff90c150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0b0800 con 0x7ff90c15e730 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0afb80 con 0x7ff90c15a4e0 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff913790640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff90c0afe60 con 0x7ff90c143490 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40024/0 (socket says 172.21.15.124:40024) 2023-12-13T23:55:47.608 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 -- 172.21.15.124:0/3200535017 learned_addr learned my addr 172.21.15.124:0/3200535017 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff910d2b640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 0x7ff90c150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff91152c640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c143490 0x7ff90c0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 -- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c143490 msgr2=0x7ff90c0af330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c143490 0x7ff90c0af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 -- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 msgr2=0x7ff90c150690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 0x7ff90c150690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff911d2d640 1 -- 172.21.15.124:0/3200535017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90800bd40 con 0x7ff90c15e730 2023-12-13T23:55:47.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.607+0000 7ff910d2b640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 0x7ff90c150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:47.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff911d2d640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff90400c7f0 tx=0x7ff90400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904007820 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff904007980 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff90c0b0140 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff904007d70 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.608+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90c0af870 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.609+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff90c0b0940 con 0x7ff90c15e730 2023-12-13T23:55:47.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.609+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff9040113f0 con 0x7ff90c15e730 2023-12-13T23:55:47.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.609+0000 7ff901ffb640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 0x7ff8f006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.610+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff904089e00 con 0x7ff90c15e730 2023-12-13T23:55:47.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.610+0000 7ff91152c640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 0x7ff8f006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.610+0000 7ff91152c640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 0x7ff8f006d680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff908037000 tx=0x7ff908032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.614+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff90c0b0940 con 0x7ff90c15e730 2023-12-13T23:55:47.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.780+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff90c0b0940 con 0x7ff90c15e730 2023-12-13T23:55:47.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.781+0000 7ff901ffb640 1 -- 172.21.15.124:0/3200535017 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff90401f020 con 0x7ff90c15e730 2023-12-13T23:55:47.779 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:47.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.782+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 msgr2=0x7ff8f006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.782+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 0x7ff8f006d680 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff908037000 tx=0x7ff908032000 comp rx=0 tx=0).stop 2023-12-13T23:55:47.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.782+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 msgr2=0x7ff90c1402d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.782+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff90400c7f0 tx=0x7ff90400ccb0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 shutdown_connections 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff8f006b1c0 0x7ff8f006d680 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff90c15a4e0 0x7ff90c150690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff90c15e730 0x7ff90c1402d0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 --2- 172.21.15.124:0/3200535017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff90c143490 0x7ff90c0af330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 >> 172.21.15.124:0/3200535017 conn(0x7ff90c06a4d0 msgr2=0x7ff90c071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 shutdown_connections 2023-12-13T23:55:47.782 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.783+0000 7ff913790640 1 -- 172.21.15.124:0/3200535017 wait complete. 2023-12-13T23:55:47.792 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:47.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.897+0000 7fab3402b640 1 Processor -- start 2023-12-13T23:55:47.896 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 -- start start 2023-12-13T23:55:47.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fab2c0d2910 0x7fab2c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fab2c0d3730 0x7fab2c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2c0b2b10 con 0x7fab2c15d330 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c0d3bc0 con 0x7fab2c0d2910 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.898+0000 7fab3402b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2c0d1590 con 0x7fab2c0d3730 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fab2c0d2910 0x7fab2c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fab2c0d2910 0x7fab2c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40036/0 (socket says 172.21.15.124:40036) 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 -- 172.21.15.124:0/4063053306 learned_addr learned my addr 172.21.15.124:0/4063053306 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab325c8640 1 --2- 172.21.15.124:0/4063053306 >> v2:172.21.15.203:3300/0 conn(0x7fab2c0d3730 0x7fab2c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 -- 172.21.15.124:0/4063053306 >> v2:172.21.15.203:3300/0 conn(0x7fab2c0d3730 msgr2=0x7fab2c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 --2- 172.21.15.124:0/4063053306 >> v2:172.21.15.203:3300/0 conn(0x7fab2c0d3730 0x7fab2c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 -- 172.21.15.124:0/4063053306 >> v1:172.21.15.203:6789/0 conn(0x7fab2c15d330 legacy=0x7fab2c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.899+0000 7fab315c6640 1 -- 172.21.15.124:0/4063053306 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0d1870 con 0x7fab2c0d2910 2023-12-13T23:55:47.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab315c6640 1 --2- 172.21.15.124:0/4063053306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c0d2910 0x7fab2c0d31f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fab2c06c500 tx=0x7fab2802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5debecbf6059d4cb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab30dc5640 1 -- 172.21.15.124:0/4063053306 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28031870 con 0x7fab2c0d2910 2023-12-13T23:55:47.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab30dc5640 1 -- 172.21.15.124:0/4063053306 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab280319d0 con 0x7fab2c0d2910 2023-12-13T23:55:47.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab30dc5640 1 -- 172.21.15.124:0/4063053306 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28031d60 con 0x7fab2c0d2910 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 -- 172.21.15.124:0/4063053306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c0d2910 msgr2=0x7fab2c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 --2- 172.21.15.124:0/4063053306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c0d2910 0x7fab2c0d31f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fab2c06c500 tx=0x7fab2802b910 comp rx=0 tx=0).stop 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 -- 172.21.15.124:0/4063053306 shutdown_connections 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 --2- 172.21.15.124:0/4063053306 >> v2:172.21.15.203:3300/0 conn(0x7fab2c0d3730 0x7fab2c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 --2- 172.21.15.124:0/4063053306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c0d2910 0x7fab2c0d31f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 -- 172.21.15.124:0/4063053306 >> 172.21.15.124:0/4063053306 conn(0x7fab2c072020 msgr2=0x7fab2c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 -- 172.21.15.124:0/4063053306 shutdown_connections 2023-12-13T23:55:47.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.900+0000 7fab3402b640 1 -- 172.21.15.124:0/4063053306 wait complete. 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.901+0000 7fab3402b640 1 Processor -- start 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 -- start start 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 0x7fab2c1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 0x7fab2c0c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c7a50 con 0x7fab2c15d330 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c6ba0 con 0x7fab2c0d2910 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab3402b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab2c0c6e40 con 0x7fab2c0d3730 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab325c8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab325c8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40052/0 (socket says 172.21.15.124:40052) 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab325c8640 1 -- 172.21.15.124:0/2761749398 learned_addr learned my addr 172.21.15.124:0/2761749398 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab315c6640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 0x7fab2c0c7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.902+0000 7fab31dc7640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 0x7fab2c1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 -- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 msgr2=0x7fab2c0c7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 0x7fab2c0c7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 -- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 msgr2=0x7fab2c1f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 0x7fab2c1f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 -- 172.21.15.124:0/2761749398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab280094d0 con 0x7fab2c15d330 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab315c6640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 0x7fab2c0c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab31dc7640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 0x7fab2c1f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab325c8640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fab2400efc0 tx=0x7fab2400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24014480 con 0x7fab2c15d330 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2400f040 con 0x7fab2c15d330 2023-12-13T23:55:47.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0c7120 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24014820 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.903+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab2c0c76e0 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.904+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2c0c7bc0 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.904+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fab24014980 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.905+0000 7fab227fc640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 0x7fab1006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.905+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab2408a600 con 0x7fab2c15d330 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.905+0000 7fab31dc7640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 0x7fab1006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.905+0000 7fab31dc7640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 0x7fab1006d6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab1c007fd0 tx=0x7fab1c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:47.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:47.909+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fab24055330 con 0x7fab2c15d330 2023-12-13T23:55:48.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.076+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fab2c069400 con 0x7fab2c15d330 2023-12-13T23:55:48.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.077+0000 7fab227fc640 1 -- 172.21.15.124:0/2761749398 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7fab24058950 con 0x7fab2c15d330 2023-12-13T23:55:48.075 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:48.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 msgr2=0x7fab1006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 0x7fab1006d6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fab1c007fd0 tx=0x7fab1c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 msgr2=0x7fab2c0cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fab2400efc0 tx=0x7fab2400c490 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 shutdown_connections 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fab1006b1f0 0x7fab1006d6b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fab2c15d330 0x7fab2c0cacf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fab2c0d3730 0x7fab2c0c7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 --2- 172.21.15.124:0/2761749398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fab2c0d2910 0x7fab2c1f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 >> 172.21.15.124:0/2761749398 conn(0x7fab2c072020 msgr2=0x7fab2c0b6a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 shutdown_connections 2023-12-13T23:55:48.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.079+0000 7fab3402b640 1 -- 172.21.15.124:0/2761749398 wait complete. 2023-12-13T23:55:48.088 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:48.189 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.190+0000 7fa4bb47c640 1 Processor -- start 2023-12-13T23:55:48.189 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.190+0000 7fa4bb47c640 1 -- start start 2023-12-13T23:55:48.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4bb47c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa4b406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4bb47c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b40adae0 con 0x7fa4b415d1d0 2023-12-13T23:55:48.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4bb47c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4b40addc0 con 0x7fa4b406bbf0 2023-12-13T23:55:48.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4bb47c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40ae0a0 con 0x7fa4b40ad640 2023-12-13T23:55:48.192 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4b9218640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa4b415d1d0 0x7fa4b406b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56842/0 (socket says 172.21.15.124:56842) 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4b9218640 1 -- 172.21.15.124:0/3249024172 learned_addr learned my addr 172.21.15.124:0/3249024172 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4b8a17640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa4b406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703326194 0 0) 0x7fa4b40adae0 con 0x7fa4b415d1d0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.191+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa49c003520 con 0x7fa4b415d1d0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 709483970 0 0) 0x7fa49c003520 con 0x7fa4b415d1d0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 msgr2=0x7fa4b406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa4b406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 >> v1:172.21.15.203:6789/0 conn(0x7fa4b406bbf0 legacy=0x7fa4b406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b40ae380 con 0x7fa4b415d1d0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fa4a40019f0 con 0x7fa4b415d1d0 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 >> v1:172.21.15.124:6789/0 conn(0x7fa4b415d1d0 legacy=0x7fa4b406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa49c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.193 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa49c003f20 0x7fa49c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa49c004d00 0x7fa49c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b40ae0a0 con 0x7fa49c004d00 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c00ba60 con 0x7fa49c003f20 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49c00bc20 con 0x7fa4b40ad640 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa49c004d00 0x7fa49c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9218640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa49c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b8a17640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa49c003f20 0x7fa49c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 -- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 msgr2=0x7fa49c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b40ad640 0x7fa49c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 -- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa49c003f20 msgr2=0x7fa49c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa49c003f20 0x7fa49c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.192+0000 7fa4b9a19640 1 -- 172.21.15.124:0/3249024172 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c003520 con 0x7fa49c004d00 2023-12-13T23:55:48.194 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.193+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa49c004d00 0x7fa49c005180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40b4220 tx=0x7fa49c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15399d971e4c2a12 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.193+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0052d0 con 0x7fa49c004d00 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.193+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac02e050 con 0x7fa49c004d00 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.193+0000 7fa4b3fff640 1 -- 172.21.15.124:0/3249024172 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac003830 con 0x7fa49c004d00 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3249024172 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa49c004d00 msgr2=0x7fa49c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3249024172 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa49c004d00 0x7fa49c005180 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40b4220 tx=0x7fa49c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4b9a19640 1 -- 172.21.15.124:0/3249024172 reap_dead start 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3249024172 shutdown_connections 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3249024172 >> 172.21.15.124:0/3249024172 conn(0x7fa4b4070e70 msgr2=0x7fa4b4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3249024172 shutdown_connections 2023-12-13T23:55:48.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.194+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3249024172 wait complete. 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.195+0000 7fa4bb47c640 1 Processor -- start 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.195+0000 7fa4bb47c640 1 -- start start 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.195+0000 7fa4bb47c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 0x7fa4b406aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 0x7fa4b4141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b4142f60 con 0x7fa4b4147440 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b4141ec0 con 0x7fa4b4147060 2023-12-13T23:55:48.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b41421a0 con 0x7fa4b4141150 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40066/0 (socket says 172.21.15.124:40066) 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 -- 172.21.15.124:0/3752637132 learned_addr learned my addr 172.21.15.124:0/3752637132 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 0x7fa4b4141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b9218640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 0x7fa4b406aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 -- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 msgr2=0x7fa4b4141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 0x7fa4b4141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 -- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 msgr2=0x7fa4b406aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 0x7fa4b406aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 -- 172.21.15.124:0/3752637132 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b4142480 con 0x7fa4b4147440 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b9a19640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 0x7fa4b4141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b9218640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 0x7fa4b406aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b8a17640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80119a0 tx=0x7fa4a8011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a800faf0 con 0x7fa4b4147440 2023-12-13T23:55:48.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a800fc50 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b4142760 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a8014440 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.196+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4b4141b40 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.198+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4b4067d20 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.198+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa4a80145a0 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.198+0000 7fa4b1ffb640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 0x7fa4a006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.198+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4a808a360 con 0x7fa4b4147440 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.199+0000 7fa4b9218640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 0x7fa4a006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.199+0000 7fa4b9218640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 0x7fa4a006d680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4004bc0 tx=0x7fa4a4006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.202+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa4a8055090 con 0x7fa4b4147440 2023-12-13T23:55:48.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.380+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa4b4069400 con 0x7fa4b4147440 2023-12-13T23:55:48.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.381+0000 7fa4b1ffb640 1 -- 172.21.15.124:0/3752637132 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa4a80586b0 con 0x7fa4b4147440 2023-12-13T23:55:48.379 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:48.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 msgr2=0x7fa4a006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 0x7fa4a006d680 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4004bc0 tx=0x7fa4a4006040 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 msgr2=0x7fa4b406b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa4a80119a0 tx=0x7fa4a8011e60 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 shutdown_connections 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa4a006b1c0 0x7fa4a006d680 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4b4141150 0x7fa4b4141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4b4147440 0x7fa4b406b430 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 --2- 172.21.15.124:0/3752637132 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4b4147060 0x7fa4b406aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 >> 172.21.15.124:0/3752637132 conn(0x7fa4b4070e70 msgr2=0x7fa4b41534c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 shutdown_connections 2023-12-13T23:55:48.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.383+0000 7fa4bb47c640 1 -- 172.21.15.124:0/3752637132 wait complete. 2023-12-13T23:55:48.392 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:55:48.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:48.392 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:55:48.419 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:55:48.419 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:55:48.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:48.419 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:55:48.483 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:48.484 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:48.503 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:48.503 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:48.561 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:48.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.662+0000 7f94d103d640 1 Processor -- start 2023-12-13T23:55:48.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.662+0000 7f94d103d640 1 -- start start 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94d103d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f94cc06f630 0x7f94cc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94d103d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06ee30 0x7f94cc0b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94d103d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94cc0cf000 con 0x7f94cc143b50 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94d103d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94cc0cfde0 con 0x7f94cc06f630 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94d103d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc06f2c0 con 0x7f94cc06ee30 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f94cc06f630 0x7f94cc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f94cc06f630 0x7f94cc06e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40070/0 (socket says 172.21.15.124:40070) 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 -- 172.21.15.124:0/635267438 learned_addr learned my addr 172.21.15.124:0/635267438 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94cb59e640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06ee30 0x7f94cc0b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 -- 172.21.15.124:0/635267438 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06ee30 msgr2=0x7f94cc0b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06ee30 0x7f94cc0b33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 -- 172.21.15.124:0/635267438 >> v1:172.21.15.124:6789/0 conn(0x7f94cc143b50 legacy=0x7f94cc0ce8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.663+0000 7f94c259c640 1 -- 172.21.15.124:0/635267438 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94cc0cf260 con 0x7f94cc06f630 2023-12-13T23:55:48.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.664+0000 7f94c259c640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc06f630 0x7f94cc06e8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f94b400bb10 tx=0x7f94b402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd0a97ac90b02e1c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.664+0000 7f94ca59c640 1 -- 172.21.15.124:0/635267438 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b402ee70 con 0x7f94cc06f630 2023-12-13T23:55:48.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.664+0000 7f94ca59c640 1 -- 172.21.15.124:0/635267438 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b4036440 con 0x7f94cc06f630 2023-12-13T23:55:48.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.664+0000 7f94ca59c640 1 -- 172.21.15.124:0/635267438 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b40367d0 con 0x7f94cc06f630 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 -- 172.21.15.124:0/635267438 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc06f630 msgr2=0x7f94cc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc06f630 0x7f94cc06e8f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f94b400bb10 tx=0x7f94b402e0c0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 -- 172.21.15.124:0/635267438 shutdown_connections 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06ee30 0x7f94cc0b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 --2- 172.21.15.124:0/635267438 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc06f630 0x7f94cc06e8f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 -- 172.21.15.124:0/635267438 >> 172.21.15.124:0/635267438 conn(0x7f94cc06d3b0 msgr2=0x7f94cc06d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 -- 172.21.15.124:0/635267438 shutdown_connections 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.665+0000 7f94d103d640 1 -- 172.21.15.124:0/635267438 wait complete. 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.666+0000 7f94d103d640 1 Processor -- start 2023-12-13T23:55:48.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.666+0000 7f94d103d640 1 -- start start 2023-12-13T23:55:48.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc06ee30 0x7f94cc1396f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 0x7f94cc0719e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc1531d0 con 0x7f94cc143b50 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0c9500 con 0x7f94cc06ee30 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94d103d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94cc0c97e0 con 0x7f94cc06f630 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94cb59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94cb59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40086/0 (socket says 172.21.15.124:40086) 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94cb59e640 1 -- 172.21.15.124:0/1318447746 learned_addr learned my addr 172.21.15.124:0/1318447746 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94c259c640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 0x7f94cc0719e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.667+0000 7f94cad9d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc06ee30 0x7f94cc1396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 -- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 msgr2=0x7f94cc0719e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 0x7f94cc0719e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 -- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc06ee30 msgr2=0x7f94cc1396f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc06ee30 0x7f94cc1396f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 -- 172.21.15.124:0/1318447746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b400b770 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94c259c640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 0x7f94cc0719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94cb59e640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94c400cd70 tx=0x7f94c400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4003b90 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c4003cf0 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c4004240 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94cc0c9ac0 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.668+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94cc139c30 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.669+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94cc0c9fe0 con 0x7f94cc143b50 2023-12-13T23:55:48.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.669+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f94c40043a0 con 0x7f94cc143b50 2023-12-13T23:55:48.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.670+0000 7f94c3fff640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 0x7f94ac06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.670+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94c4012070 con 0x7f94cc143b50 2023-12-13T23:55:48.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.670+0000 7f94cad9d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 0x7f94ac06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.671+0000 7f94cad9d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 0x7f94ac06d680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94bc000f60 tx=0x7f94bc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.674+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f94cc0c9fe0 con 0x7f94cc143b50 2023-12-13T23:55:48.841 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.842+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94cc0c9fe0 con 0x7f94cc143b50 2023-12-13T23:55:48.842 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.843+0000 7f94c3fff640 1 -- 172.21.15.124:0/1318447746 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f94cc0c9fe0 con 0x7f94cc143b50 2023-12-13T23:55:48.842 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:48.842 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 msgr2=0x7f94ac06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 0x7f94ac06d680 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f94bc000f60 tx=0x7f94bc009040 comp rx=0 tx=0).stop 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 msgr2=0x7f94cc071f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f94c400cd70 tx=0x7f94c400a660 comp rx=0 tx=0).stop 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 shutdown_connections 2023-12-13T23:55:48.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f94ac06b1c0 0x7f94ac06d680 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f94cc143b50 0x7f94cc071f20 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f94cc06f630 0x7f94cc0719e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 --2- 172.21.15.124:0/1318447746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f94cc06ee30 0x7f94cc1396f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 >> 172.21.15.124:0/1318447746 conn(0x7f94cc06d3b0 msgr2=0x7f94cc15b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.845+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 shutdown_connections 2023-12-13T23:55:48.845 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.846+0000 7f94d103d640 1 -- 172.21.15.124:0/1318447746 wait complete. 2023-12-13T23:55:48.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:48.856 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:48.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 Processor -- start 2023-12-13T23:55:48.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 -- start start 2023-12-13T23:55:48.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f721815ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f721815b5d0 0x7f721815ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f721815f100 con 0x7f721806a4d0 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72180714c0 con 0x7f721815b5d0 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f721d397640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f721815bf60 con 0x7f72181448f0 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.960+0000 7f72177fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f721815b5d0 0x7f721815ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f721815b5d0 0x7f721815ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40090/0 (socket says 172.21.15.124:40090) 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 -- 172.21.15.124:0/1187038276 learned_addr learned my addr 172.21.15.124:0/1187038276 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f7216ffd640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f721815ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 -- 172.21.15.124:0/1187038276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 msgr2=0x7f721815ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f721815ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 -- 172.21.15.124:0/1187038276 >> v1:172.21.15.203:6789/0 conn(0x7f721806a4d0 legacy=0x7f7218071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.961+0000 7f72177fe640 1 -- 172.21.15.124:0/1187038276 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721815f3a0 con 0x7f721815b5d0 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f72177fe640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721815ba20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f721002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baf56b4020d5cf1c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f7215ffb640 1 -- 172.21.15.124:0/1187038276 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210030840 con 0x7f721815b5d0 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f7215ffb640 1 -- 172.21.15.124:0/1187038276 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72100309a0 con 0x7f721815b5d0 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f7215ffb640 1 -- 172.21.15.124:0/1187038276 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210030d30 con 0x7f721815b5d0 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 -- 172.21.15.124:0/1187038276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 msgr2=0x7f721815ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721815ba20 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f721002b980 comp rx=0 tx=0).stop 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 -- 172.21.15.124:0/1187038276 shutdown_connections 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721815ba20 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 --2- 172.21.15.124:0/1187038276 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f721815ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 -- 172.21.15.124:0/1187038276 >> 172.21.15.124:0/1187038276 conn(0x7f72180b1de0 msgr2=0x7f72180b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:48.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 -- 172.21.15.124:0/1187038276 shutdown_connections 2023-12-13T23:55:48.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.962+0000 7f721d397640 1 -- 172.21.15.124:0/1187038276 wait complete. 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 Processor -- start 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 -- start start 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f721806a4d0 0x7f721813f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f72180cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721813fbb0 con 0x7f721815b5d0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72180c54e0 con 0x7f721806a4d0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f721d397640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72180c57c0 con 0x7f72181448f0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f72177fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f72177fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40102/0 (socket says 172.21.15.124:40102) 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.964+0000 7f72177fe640 1 -- 172.21.15.124:0/990386914 learned_addr learned my addr 172.21.15.124:0/990386914 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f7216ffd640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f721806a4d0 0x7f721813f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72167fc640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f72180cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 -- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 msgr2=0x7f72180cbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f72180cbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 -- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f721806a4d0 msgr2=0x7f721813f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f721806a4d0 0x7f721813f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 -- 172.21.15.124:0/990386914 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72100094d0 con 0x7f721815b5d0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f72177fe640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f7210004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210039c90 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7210039e10 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72180c5aa0 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7210043640 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.965+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72180c6060 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.966+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72180c6250 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.966+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f72100437a0 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.967+0000 7f71ff7fe640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 0x7f720406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.967+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f721003e070 con 0x7f721815b5d0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.967+0000 7f7216ffd640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 0x7f720406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:48.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.968+0000 7f7216ffd640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 0x7f720406d680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7208009f60 tx=0x7f720800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:48.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:48.971+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7210077890 con 0x7f721815b5d0 2023-12-13T23:55:49.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.141+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7218067420 con 0x7f721815b5d0 2023-12-13T23:55:49.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.141+0000 7f71ff7fe640 1 -- 172.21.15.124:0/990386914 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f72100340a0 con 0x7f721815b5d0 2023-12-13T23:55:49.140 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 msgr2=0x7f720406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 0x7f720406d680 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7208009f60 tx=0x7f720800b040 comp rx=0 tx=0).stop 2023-12-13T23:55:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 msgr2=0x7f721806d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7210009870 tx=0x7f7210004730 comp rx=0 tx=0).stop 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 shutdown_connections 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f720406b1c0 0x7f720406d680 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f721815b5d0 0x7f721806d5b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f72181448f0 0x7f72180cbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 --2- 172.21.15.124:0/990386914 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f721806a4d0 0x7f721813f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 >> 172.21.15.124:0/990386914 conn(0x7f72180b1de0 msgr2=0x7f72180b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 shutdown_connections 2023-12-13T23:55:49.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.143+0000 7f721d397640 1 -- 172.21.15.124:0/990386914 wait complete. 2023-12-13T23:55:49.152 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:49.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.256+0000 7ffbaf33c640 1 Processor -- start 2023-12-13T23:55:49.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.257+0000 7ffbaf33c640 1 -- start start 2023-12-13T23:55:49.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.257+0000 7ffbaf33c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbaf33c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba815e2a0 con 0x7ffba815e730 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbaf33c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba815b240 con 0x7ffba8143490 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbaf33c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba815b520 con 0x7ffba80b1400 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbad8d9640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ffba815e730 0x7ffba815aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56874/0 (socket says 172.21.15.124:56874) 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbad8d9640 1 -- 172.21.15.124:0/872502622 learned_addr learned my addr 172.21.15.124:0/872502622 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbac8d7640 1 --2- 172.21.15.124:0/872502622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287699701 0 0) 0x7ffba815e2a0 con 0x7ffba815e730 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb94003520 con 0x7ffba815e730 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249114466 0 0) 0x7ffba815b240 con 0x7ffba8143490 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba815e2a0 con 0x7ffba8143490 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbac8d7640 1 -- 172.21.15.124:0/872502622 >> v1:172.21.15.203:6789/0 conn(0x7ffba8143490 legacy=0x7ffba80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbac8d7640 1 -- 172.21.15.124:0/872502622 >> v1:172.21.15.124:6789/0 conn(0x7ffba815e730 legacy=0x7ffba815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884787594 0 0) 0x7ffb94003520 con 0x7ffba815e730 2023-12-13T23:55:49.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.258+0000 7ffbac8d7640 1 -- 172.21.15.124:0/872502622 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba815dc50 con 0x7ffba80b1400 2023-12-13T23:55:49.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.259+0000 7ffbac8d7640 1 --2- 172.21.15.124:0/872502622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba815a4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffb980097d0 tx=0x7ffb9802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=179ee4fca31c0204 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.259+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba00019f0 con 0x7ffba80b1400 2023-12-13T23:55:49.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.259+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9802e050 con 0x7ffba80b1400 2023-12-13T23:55:49.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.259+0000 7ffb9ffff640 1 -- 172.21.15.124:0/872502622 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98030730 con 0x7ffba80b1400 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 -- 172.21.15.124:0/872502622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 msgr2=0x7ffba815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/872502622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba815a4e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ffb980097d0 tx=0x7ffb9802b800 comp rx=0 tx=0).stop 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 -- 172.21.15.124:0/872502622 shutdown_connections 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/872502622 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba815a4e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 -- 172.21.15.124:0/872502622 >> 172.21.15.124:0/872502622 conn(0x7ffba806a4d0 msgr2=0x7ffba806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 -- 172.21.15.124:0/872502622 shutdown_connections 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.260+0000 7ffbaf33c640 1 -- 172.21.15.124:0/872502622 wait complete. 2023-12-13T23:55:49.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.261+0000 7ffbaf33c640 1 Processor -- start 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.261+0000 7ffbaf33c640 1 -- start start 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba80b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 0x7ffba81368a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba80b1200 con 0x7ffba8143490 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba8159af0 con 0x7ffba815e730 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbaf33c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba813a7e0 con 0x7ffba80b1400 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40114/0 (socket says 172.21.15.124:40114) 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 -- 172.21.15.124:0/2794113830 learned_addr learned my addr 172.21.15.124:0/2794113830 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbad0d8640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba80b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbad8d9640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 0x7ffba81368a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 -- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 msgr2=0x7ffba80b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba80b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 -- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 msgr2=0x7ffba81368a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 0x7ffba81368a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 -- 172.21.15.124:0/2794113830 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb98009400 con 0x7ffba8143490 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbad8d9640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 0x7ffba81368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffbac8d7640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000c00 tx=0x7ffb980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98004230 con 0x7ffba8143490 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.262+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb980043b0 con 0x7ffba8143490 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.263+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba813aa80 con 0x7ffba8143490 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.263+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98004520 con 0x7ffba8143490 2023-12-13T23:55:49.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.263+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba813b040 con 0x7ffba8143490 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.264+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba80ce940 con 0x7ffba8143490 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.264+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ffb98039040 con 0x7ffba8143490 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.264+0000 7ffb9dffb640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 0x7ffb8c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.265+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffb980aae30 con 0x7ffba8143490 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.265+0000 7ffbad0d8640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 0x7ffb8c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.265+0000 7ffbad0d8640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 0x7ffb8c06d6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffba40011e0 tx=0x7ffba4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.269+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ffb98075dc0 con 0x7ffba8143490 2023-12-13T23:55:49.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.434+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffba813b230 con 0x7ffba8143490 2023-12-13T23:55:49.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.435+0000 7ffb9dffb640 1 -- 172.21.15.124:0/2794113830 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb98037020 con 0x7ffba8143490 2023-12-13T23:55:49.433 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:49.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 msgr2=0x7ffb8c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 0x7ffb8c06d6b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ffba40011e0 tx=0x7ffba4009040 comp rx=0 tx=0).stop 2023-12-13T23:55:49.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 msgr2=0x7ffba815d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000c00 tx=0x7ffb980046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 shutdown_connections 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8c06b1f0 0x7ffb8c06d6b0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba815e730 0x7ffba81368a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba8143490 0x7ffba815d300 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 --2- 172.21.15.124:0/2794113830 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba80b1400 0x7ffba80b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 >> 172.21.15.124:0/2794113830 conn(0x7ffba806a4d0 msgr2=0x7ffba80ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 shutdown_connections 2023-12-13T23:55:49.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.437+0000 7ffbaf33c640 1 -- 172.21.15.124:0/2794113830 wait complete. 2023-12-13T23:55:49.447 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.551+0000 7f1580a44640 1 Processor -- start 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.551+0000 7f1580a44640 1 -- start start 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f1580a44640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f1580a44640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f157c15e730 0x7f157c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f1580a44640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f157c15e2a0 con 0x7f157c143490 2023-12-13T23:55:49.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f1580a44640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157c15b520 con 0x7f157c15e730 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f1580a44640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c143870 con 0x7f157c0b1400 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f157ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f157c15e730 0x7f157c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f157ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f157c15e730 0x7f157c15aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40130/0 (socket says 172.21.15.124:40130) 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f157ad9d640 1 -- 172.21.15.124:0/4245064847 learned_addr learned my addr 172.21.15.124:0/4245064847 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.552+0000 7f157a59c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f157c143490 0x7f157c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56878/0 (socket says 172.21.15.124:56878) 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f1579d9b640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157ad9d640 1 -- 172.21.15.124:0/4245064847 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 msgr2=0x7f157c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157ad9d640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c15a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157ad9d640 1 -- 172.21.15.124:0/4245064847 >> v1:172.21.15.124:6789/0 conn(0x7f157c143490 legacy=0x7f157c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157ad9d640 1 -- 172.21.15.124:0/4245064847 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c15b240 con 0x7f157c15e730 2023-12-13T23:55:49.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157ad9d640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c15aa20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1574009870 tx=0x7f157402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29bca28197afec0b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157959a640 1 -- 172.21.15.124:0/4245064847 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574030840 con 0x7f157c15e730 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.553+0000 7f157959a640 1 -- 172.21.15.124:0/4245064847 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15740309a0 con 0x7f157c15e730 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f157959a640 1 -- 172.21.15.124:0/4245064847 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574030d30 con 0x7f157c15e730 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 -- 172.21.15.124:0/4245064847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 msgr2=0x7f157c15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c15aa20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1574009870 tx=0x7f157402b980 comp rx=0 tx=0).stop 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 -- 172.21.15.124:0/4245064847 shutdown_connections 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 --2- 172.21.15.124:0/4245064847 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c15aa20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 -- 172.21.15.124:0/4245064847 >> 172.21.15.124:0/4245064847 conn(0x7f157c06a4d0 msgr2=0x7f157c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 -- 172.21.15.124:0/4245064847 shutdown_connections 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.554+0000 7f1580a44640 1 -- 172.21.15.124:0/4245064847 wait complete. 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.555+0000 7f1580a44640 1 Processor -- start 2023-12-13T23:55:49.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.555+0000 7f1580a44640 1 -- start start 2023-12-13T23:55:49.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 0x7f157c13c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c15a050 con 0x7f157c15e730 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157c1412f0 con 0x7f157c143490 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1580a44640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157c1406e0 con 0x7f157c0b1400 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f157ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f157ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40132/0 (socket says 172.21.15.124:40132) 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f157ad9d640 1 -- 172.21.15.124:0/386964213 learned_addr learned my addr 172.21.15.124:0/386964213 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f157a59c640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.556+0000 7f1579d9b640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 0x7f157c13c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 -- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 msgr2=0x7f157c159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 -- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 msgr2=0x7f157c13c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 0x7f157c13c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 -- 172.21.15.124:0/386964213 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740094d0 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157a59c640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f1579d9b640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 0x7f157c13c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f157ad9d640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1574009870 tx=0x7f1574004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574039a90 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1574039c10 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574043410 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157c1409c0 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.557+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157c140f80 con 0x7f157c15e730 2023-12-13T23:55:49.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.558+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157c0b4380 con 0x7f157c15e730 2023-12-13T23:55:49.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.558+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f157402f390 con 0x7f157c15e730 2023-12-13T23:55:49.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.559+0000 7f156affd640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 0x7f155c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.559+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15740abbd0 con 0x7f157c15e730 2023-12-13T23:55:49.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.559+0000 7f157a59c640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 0x7f155c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.559+0000 7f157a59c640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 0x7f155c06d680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f156c007fd0 tx=0x7f156c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.563+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1574076900 con 0x7f157c15e730 2023-12-13T23:55:49.730 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.731+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f157c069400 con 0x7f157c15e730 2023-12-13T23:55:49.730 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.732+0000 7f156affd640 1 -- 172.21.15.124:0/386964213 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f15740480a0 con 0x7f157c15e730 2023-12-13T23:55:49.730 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:49.732 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 msgr2=0x7f155c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.732 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 0x7f155c06d680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f156c007fd0 tx=0x7f156c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:49.732 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 msgr2=0x7f157c13cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1574009870 tx=0x7f1574004730 comp rx=0 tx=0).stop 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 shutdown_connections 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f155c06b1c0 0x7f155c06d680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f157c143490 0x7f157c13c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f157c15e730 0x7f157c13cba0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 --2- 172.21.15.124:0/386964213 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f157c0b1400 0x7f157c159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 >> 172.21.15.124:0/386964213 conn(0x7f157c06a4d0 msgr2=0x7f157c0aedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 shutdown_connections 2023-12-13T23:55:49.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.733+0000 7f1580a44640 1 -- 172.21.15.124:0/386964213 wait complete. 2023-12-13T23:55:49.743 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:49.843 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.845+0000 7f81aa916640 1 Processor -- start 2023-12-13T23:55:49.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.845+0000 7f81aa916640 1 -- start start 2023-12-13T23:55:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81aa916640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f81a406abe0 0x7f81a41618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81aa916640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a41406f0 con 0x7f81a4140310 2023-12-13T23:55:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81aa916640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a4143960 con 0x7f81a41409d0 2023-12-13T23:55:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81aa916640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81a4161e20 con 0x7f81a406abe0 2023-12-13T23:55:49.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f81a4140310 0x7f81a406a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56890/0 (socket says 172.21.15.124:56890) 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a3fff640 1 -- 172.21.15.124:0/612731278 learned_addr learned my addr 172.21.15.124:0/612731278 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a37fe640 1 --2- 172.21.15.124:0/612731278 >> v2:172.21.15.203:3300/0 conn(0x7f81a406abe0 0x7f81a41618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846545157 0 0) 0x7f81a41406f0 con 0x7f81a4140310 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c003520 con 0x7f81a4140310 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3747470944 0 0) 0x7f818c003520 con 0x7f81a4140310 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 >> v2:172.21.15.203:3300/0 conn(0x7f81a406abe0 msgr2=0x7f81a41618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 --2- 172.21.15.124:0/612731278 >> v2:172.21.15.203:3300/0 conn(0x7f81a406abe0 0x7f81a41618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 >> v1:172.21.15.203:6789/0 conn(0x7f81a41409d0 legacy=0x7f81a406efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4142de0 con 0x7f81a4140310 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8198001670 con 0x7f81a4140310 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 >> v1:172.21.15.124:6789/0 conn(0x7f81a4140310 legacy=0x7f81a406a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f818c003970 0x7f818c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f818c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c005180 0x7f818c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4161e20 con 0x7f818c0047c0 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f818c00c120 con 0x7f818c005180 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f818c00c2e0 con 0x7f818c003970 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.846+0000 7f81a37fe640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f818c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a8eb3640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c005180 0x7f818c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a3fff640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f818c003970 0x7f818c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 -- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f818c003970 msgr2=0x7f818c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f818c003970 0x7f818c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 -- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c005180 msgr2=0x7f818c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c005180 0x7f818c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 -- 172.21.15.124:0/612731278 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c003520 con 0x7f818c0047c0 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a8eb3640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f818c005180 0x7f818c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:49.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a8eb3640 1 -- 172.21.15.124:0/612731278 reap_dead start 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.847+0000 7f81a37fe640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f818c004c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8194009ac0 tx=0x7f818c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3835584b8dcecf server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819402c6b0 con 0x7f818c0047c0 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819402c810 con 0x7f818c0047c0 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81a2ffd640 1 -- 172.21.15.124:0/612731278 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819402cba0 con 0x7f818c0047c0 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 -- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 msgr2=0x7f818c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f818c004c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8194009ac0 tx=0x7f818c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 -- 172.21.15.124:0/612731278 shutdown_connections 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 --2- 172.21.15.124:0/612731278 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f818c004c40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 -- 172.21.15.124:0/612731278 >> 172.21.15.124:0/612731278 conn(0x7f81a40b1e50 msgr2=0x7f81a40b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 -- 172.21.15.124:0/612731278 shutdown_connections 2023-12-13T23:55:49.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.848+0000 7f81aa916640 1 -- 172.21.15.124:0/612731278 wait complete. 2023-12-13T23:55:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.849+0000 7f81aa916640 1 Processor -- start 2023-12-13T23:55:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 -- start start 2023-12-13T23:55:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81a413e370 0x7f81a413e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a41f6560 0x7f81a41f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a4144580 con 0x7f818c0047c0 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81a41f72c0 con 0x7f81a41f6560 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81aa916640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81a413ed30 con 0x7f81a413e370 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81a3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81a3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40154/0 (socket says 172.21.15.124:40154) 2023-12-13T23:55:49.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.850+0000 7f81a3fff640 1 -- 172.21.15.124:0/426345388 learned_addr learned my addr 172.21.15.124:0/426345388 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 -- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81a413e370 msgr2=0x7f81a413e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81a413e370 0x7f81a413e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 -- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a41f6560 msgr2=0x7f81a41f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a41f6560 0x7f81a41f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 -- 172.21.15.124:0/426345388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a41f2cb0 con 0x7f818c0047c0 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a3fff640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f819800e8f0 tx=0x7f819800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819800c7f0 con 0x7f818c0047c0 2023-12-13T23:55:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f819800c950 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a41f3cd0 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819800cd00 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.851+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a41f2fc0 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.852+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a4161030 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.852+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f819801d460 con 0x7f818c0047c0 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.853+0000 7f81a0ff9640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 0x7f819006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.853+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8198089a70 con 0x7f818c0047c0 2023-12-13T23:55:49.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.853+0000 7f81a37fe640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 0x7f819006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:49.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.853+0000 7f81a37fe640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 0x7f819006d680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8194036000 tx=0x7f819402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:49.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:49.857+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f81980547a0 con 0x7f818c0047c0 2023-12-13T23:55:50.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.038+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f81a4069400 con 0x7f818c0047c0 2023-12-13T23:55:50.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.038+0000 7f81a0ff9640 1 -- 172.21.15.124:0/426345388 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f819800ce60 con 0x7f818c0047c0 2023-12-13T23:55:50.037 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:50.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 msgr2=0x7f819006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 0x7f819006d680 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8194036000 tx=0x7f819402e000 comp rx=0 tx=0).stop 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 msgr2=0x7f81a41543b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f819800e8f0 tx=0x7f819800edb0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 shutdown_connections 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f819006b1c0 0x7f819006d680 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81a41f6560 0x7f81a41f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81a413e370 0x7f81a413e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 --2- 172.21.15.124:0/426345388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f818c0047c0 0x7f81a41543b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.040+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 >> 172.21.15.124:0/426345388 conn(0x7f81a40b1e50 msgr2=0x7f81a4073440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.041+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 shutdown_connections 2023-12-13T23:55:50.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.041+0000 7f81aa916640 1 -- 172.21.15.124:0/426345388 wait complete. 2023-12-13T23:55:50.050 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:50.050 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:50.070 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:50.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 Processor -- start 2023-12-13T23:55:50.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 -- start start 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5dc40d2da0 0x7f5dc40d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc40d12f0 con 0x7f5dc415d3f0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc415d7d0 con 0x7f5dc40b4c60 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.172+0000 7f5dc9623640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc40b61e0 con 0x7f5dc40d2da0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc2ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f5dc415d3f0 0x7f5dc40d2690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:56906/0 (socket says 172.21.15.124:56906) 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc2ffd640 1 -- 172.21.15.124:0/438458486 learned_addr learned my addr 172.21.15.124:0/438458486 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dbbfff640 1 --2- 172.21.15.124:0/438458486 >> v2:172.21.15.203:3300/0 conn(0x7f5dc40d2da0 0x7f5dc40d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716172494 0 0) 0x7f5dc40d12f0 con 0x7f5dc415d3f0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db0003520 con 0x7f5dc415d3f0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3316451600 0 0) 0x7f5db0003520 con 0x7f5dc415d3f0 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 >> v2:172.21.15.203:3300/0 conn(0x7f5dc40d2da0 msgr2=0x7f5dc40d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 --2- 172.21.15.124:0/438458486 >> v2:172.21.15.203:3300/0 conn(0x7f5dc40d2da0 0x7f5dc40d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 >> v1:172.21.15.203:6789/0 conn(0x7f5dc40b4c60 legacy=0x7f5dc40b6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40d1520 con 0x7f5dc415d3f0 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f5db4001670 con 0x7f5dc415d3f0 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 >> v1:172.21.15.124:6789/0 conn(0x7f5dc415d3f0 legacy=0x7f5dc40d2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0003970 0x7f5db0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5db00047c0 0x7f5db0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 0x7f5db000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40b61e0 con 0x7f5db0005180 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5db000c030 con 0x7f5db0003970 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5db000c1f0 con 0x7f5db00047c0 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc37fe640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 0x7f5db000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dc2ffd640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0003970 0x7f5db0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.173+0000 7f5dbbfff640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5db00047c0 0x7f5db0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 -- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5db00047c0 msgr2=0x7f5db0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5db00047c0 0x7f5db0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 -- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0003970 msgr2=0x7f5db0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0003970 0x7f5db0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 -- 172.21.15.124:0/438458486 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5db0003520 con 0x7f5db0005180 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 -- 172.21.15.124:0/438458486 reap_dead start 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc37fe640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 0x7f5db000b980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc001e50 tx=0x7f5db00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d862a40a867838fb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc003b70 con 0x7f5db0005180 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc003d10 con 0x7f5db0005180 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.174+0000 7f5dc27fc640 1 -- 172.21.15.124:0/438458486 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc005310 con 0x7f5db0005180 2023-12-13T23:55:50.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 -- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 msgr2=0x7f5db000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 0x7f5db000b980 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc001e50 tx=0x7f5db00057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 -- 172.21.15.124:0/438458486 shutdown_connections 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 --2- 172.21.15.124:0/438458486 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5db0005180 0x7f5db000b980 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 -- 172.21.15.124:0/438458486 >> 172.21.15.124:0/438458486 conn(0x7f5dc4072230 msgr2=0x7f5dc40cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 -- 172.21.15.124:0/438458486 shutdown_connections 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.175+0000 7f5dc9623640 1 -- 172.21.15.124:0/438458486 wait complete. 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.176+0000 7f5dc9623640 1 Processor -- start 2023-12-13T23:55:50.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.176+0000 7f5dc9623640 1 -- start start 2023-12-13T23:55:50.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 0x7f5dc41550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc40b7920 0x7f5dc40b7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415fc00 con 0x7f5dc415e370 2023-12-13T23:55:50.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc40b82e0 con 0x7f5db0005180 2023-12-13T23:55:50.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc9623640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc415eb60 con 0x7f5dc40b7920 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:40168/0 (socket says 172.21.15.124:40168) 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 -- 172.21.15.124:0/2664184965 learned_addr learned my addr 172.21.15.124:0/2664184965 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc2ffd640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 0x7f5dc41550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 -- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc40b7920 msgr2=0x7f5dc40b7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc37fe640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc40b7920 0x7f5dc40b7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc40b7920 0x7f5dc40b7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 -- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 msgr2=0x7f5dc41550a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 0x7f5dc41550a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dbbfff640 1 -- 172.21.15.124:0/2664184965 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc415ee40 con 0x7f5dc415e370 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.177+0000 7f5dc2ffd640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 0x7f5dc41550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dbbfff640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0118e0 tx=0x7f5dac011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac00fa90 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dac00fbf0 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc415f120 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dac014440 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.178+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc415f6e0 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.179+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc4069400 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.179+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5dac0145a0 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.179+0000 7f5dbb7fe640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 0x7f5da406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.180+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5dac08a1f0 con 0x7f5dc415e370 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.180+0000 7f5dc2ffd640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 0x7f5da406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.180+0000 7f5dc2ffd640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 0x7f5da406d680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5db40017a0 tx=0x7f5db4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.184+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5dac054fb0 con 0x7f5dc415e370 2023-12-13T23:55:50.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.351+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc40d36b0 con 0x7f5dc415e370 2023-12-13T23:55:50.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.352+0000 7f5dbb7fe640 1 -- 172.21.15.124:0/2664184965 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f5dac0585d0 con 0x7f5dc415e370 2023-12-13T23:55:50.351 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:50.351 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 msgr2=0x7f5da406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 0x7f5da406d680 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5db40017a0 tx=0x7f5db4009040 comp rx=0 tx=0).stop 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 msgr2=0x7f5dc414dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0118e0 tx=0x7f5dac011da0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 shutdown_connections 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5da406b1c0 0x7f5da406d680 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5dc40b7920 0x7f5dc40b7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dc415e370 0x7f5dc414dfb0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 --2- 172.21.15.124:0/2664184965 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5db0005180 0x7f5dc41550a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 >> 172.21.15.124:0/2664184965 conn(0x7f5dc4072230 msgr2=0x7f5dc4152700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 shutdown_connections 2023-12-13T23:55:50.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.354+0000 7f5dc9623640 1 -- 172.21.15.124:0/2664184965 wait complete. 2023-12-13T23:55:50.366 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:50.366 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:50.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.470+0000 7f833541e640 1 Processor -- start 2023-12-13T23:55:50.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 -- start start 2023-12-13T23:55:50.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f83300b1400 0x7f833015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 0x7f833015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833015e2a0 con 0x7f8330143490 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833015b520 con 0x7f83300b1400 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f833541e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8330143870 con 0x7f833015e730 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f832effd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8330143490 0x7f83300b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40756/0 (socket says 172.21.15.124:40756) 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f832effd640 1 -- 172.21.15.124:0/1568250817 learned_addr learned my addr 172.21.15.124:0/1568250817 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.471+0000 7f832e7fc640 1 --2- 172.21.15.124:0/1568250817 >> v2:172.21.15.124:3300/0 conn(0x7f83300b1400 0x7f833015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832f7fe640 1 --2- 172.21.15.124:0/1568250817 >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 0x7f833015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832e7fc640 1 -- 172.21.15.124:0/1568250817 >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 msgr2=0x7f833015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832e7fc640 1 --2- 172.21.15.124:0/1568250817 >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 0x7f833015aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832e7fc640 1 -- 172.21.15.124:0/1568250817 >> v1:172.21.15.124:6789/0 conn(0x7f8330143490 legacy=0x7f83300b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832e7fc640 1 -- 172.21.15.124:0/1568250817 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833015b240 con 0x7f83300b1400 2023-12-13T23:55:50.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832f7fe640 1 --2- 172.21.15.124:0/1568250817 >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 0x7f833015aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.472+0000 7f832e7fc640 1 --2- 172.21.15.124:0/1568250817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f833015a4e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f83300cef60 tx=0x7f832402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d3d61bb7cd4198 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f832dffb640 1 -- 172.21.15.124:0/1568250817 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324031830 con 0x7f83300b1400 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f832dffb640 1 -- 172.21.15.124:0/1568250817 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324031990 con 0x7f83300b1400 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f832dffb640 1 -- 172.21.15.124:0/1568250817 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324031d20 con 0x7f83300b1400 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 -- 172.21.15.124:0/1568250817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 msgr2=0x7f833015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 --2- 172.21.15.124:0/1568250817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f833015a4e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f83300cef60 tx=0x7f832402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 -- 172.21.15.124:0/1568250817 shutdown_connections 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 --2- 172.21.15.124:0/1568250817 >> v2:172.21.15.203:3300/0 conn(0x7f833015e730 0x7f833015aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 --2- 172.21.15.124:0/1568250817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f833015a4e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 -- 172.21.15.124:0/1568250817 >> 172.21.15.124:0/1568250817 conn(0x7f833006a4d0 msgr2=0x7f833006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 -- 172.21.15.124:0/1568250817 shutdown_connections 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.473+0000 7f833541e640 1 -- 172.21.15.124:0/1568250817 wait complete. 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 Processor -- start 2023-12-13T23:55:50.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 -- start start 2023-12-13T23:55:50.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 0x7f83300aa940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f833015e730 0x7f8330070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833012fa90 con 0x7f83300b1400 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300addf0 con 0x7f8330143490 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f833541e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833012fce0 con 0x7f833015e730 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f832effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f832effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43284/0 (socket says 172.21.15.124:43284) 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f832effd640 1 -- 172.21.15.124:0/2900602839 learned_addr learned my addr 172.21.15.124:0/2900602839 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.475+0000 7f832e7fc640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 0x7f83300aa940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832f7fe640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f833015e730 0x7f8330070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 -- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f833015e730 msgr2=0x7f8330070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f833015e730 0x7f8330070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 -- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 msgr2=0x7f83300aa940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 0x7f83300aa940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 -- 172.21.15.124:0/2900602839 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8324009400 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832e7fc640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 0x7f83300aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f832effd640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f832000cba0 tx=0x7f832000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000ad40 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320012400 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833012ffc0 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000aea0 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.476+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8330130580 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.477+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8330069400 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.477+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8320007070 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.478+0000 7f83177fe640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 0x7f831c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.478+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8320089820 con 0x7f83300b1400 2023-12-13T23:55:50.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.478+0000 7f832e7fc640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 0x7f831c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.478+0000 7f832e7fc640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 0x7f831c06d6b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8324036000 tx=0x7f832403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.482+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8320054690 con 0x7f83300b1400 2023-12-13T23:55:50.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.648+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f83300b2820 con 0x7f83300b1400 2023-12-13T23:55:50.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.648+0000 7f83177fe640 1 -- 172.21.15.124:0/2900602839 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8320017020 con 0x7f83300b1400 2023-12-13T23:55:50.647 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 msgr2=0x7f831c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 0x7f831c06d6b0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8324036000 tx=0x7f832403c040 comp rx=0 tx=0).stop 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 msgr2=0x7f83300ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f832000cba0 tx=0x7f832000a430 comp rx=0 tx=0).stop 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 shutdown_connections 2023-12-13T23:55:50.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f831c06b1f0 0x7f831c06d6b0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8330143490 0x7f83300aa940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f833015e730 0x7f8330070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 --2- 172.21.15.124:0/2900602839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83300b1400 0x7f83300ad8b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 >> 172.21.15.124:0/2900602839 conn(0x7f833006a4d0 msgr2=0x7f833006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 shutdown_connections 2023-12-13T23:55:50.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.650+0000 7f833541e640 1 -- 172.21.15.124:0/2900602839 wait complete. 2023-12-13T23:55:50.660 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:50.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.763+0000 7f4cccb5a640 1 Processor -- start 2023-12-13T23:55:50.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.763+0000 7f4cccb5a640 1 -- start start 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cccb5a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc81409d0 0x7f4cc81618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cccb5a640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc81406f0 con 0x7f4cc806abe0 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cccb5a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cc8143960 con 0x7f4cc8140310 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cccb5a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8161e20 con 0x7f4cc81409d0 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cc5d9b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f4cc806abe0 0x7f4cc806efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40764/0 (socket says 172.21.15.124:40764) 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cc5d9b640 1 -- 172.21.15.124:0/2792217824 learned_addr learned my addr 172.21.15.124:0/2792217824 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.764+0000 7f4cc6d9d640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc81409d0 0x7f4cc81618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517020567 0 0) 0x7f4cc81406f0 con 0x7f4cc806abe0 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cb0003520 con 0x7f4cc806abe0 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455946579 0 0) 0x7f4cc8143960 con 0x7f4cc8140310 2023-12-13T23:55:50.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cc81406f0 con 0x7f4cc8140310 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1378194933 0 0) 0x7f4cb0003520 con 0x7f4cc806abe0 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc81409d0 msgr2=0x7f4cc81618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc81409d0 0x7f4cc81618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 >> v1:172.21.15.203:6789/0 conn(0x7f4cc8140310 legacy=0x7f4cc806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8142de0 con 0x7f4cc806abe0 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f4cbc001670 con 0x7f4cc806abe0 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 >> v1:172.21.15.124:6789/0 conn(0x7f4cc806abe0 legacy=0x7f4cc806efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cb0003b30 0x7f4cb0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cb0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cb0005180 0x7f4cb000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8161e20 con 0x7f4cb00047c0 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb000bff0 con 0x7f4cb0003b30 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cb000c1b0 con 0x7f4cb0005180 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc6d9d640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc81409d0 0x7f4cc81618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cb0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc659c640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cb0003b30 0x7f4cb0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc6d9d640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cb0005180 0x7f4cb000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 -- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cb0005180 msgr2=0x7f4cb000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cb0005180 0x7f4cb000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 -- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cb0003b30 msgr2=0x7f4cb0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cb0003b30 0x7f4cb0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.765+0000 7f4cc5d9b640 1 -- 172.21.15.124:0/2792217824 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cb0003520 con 0x7f4cb00047c0 2023-12-13T23:55:50.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.766+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cb0004c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007fd0 tx=0x7f4cb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03b641ad56782c4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.766+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc005ad0 con 0x7f4cb00047c0 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.766+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cbc005c50 con 0x7f4cb00047c0 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.766+0000 7f4cc559a640 1 -- 172.21.15.124:0/2792217824 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cbc02c830 con 0x7f4cb00047c0 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2792217824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 msgr2=0x7f4cb0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2792217824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cb0004c40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc007fd0 tx=0x7f4cb00057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cc6d9d640 1 -- 172.21.15.124:0/2792217824 reap_dead start 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2792217824 shutdown_connections 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2792217824 >> 172.21.15.124:0/2792217824 conn(0x7f4cc80b1e50 msgr2=0x7f4cc80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2792217824 shutdown_connections 2023-12-13T23:55:50.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.767+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2792217824 wait complete. 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.768+0000 7f4cccb5a640 1 Processor -- start 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.768+0000 7f4cccb5a640 1 -- start start 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 0x7f4cc81467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8144540 0x7f4cc81449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8144f10 con 0x7f4cb00047c0 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8155af0 con 0x7f4cc8144540 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cc8154e50 con 0x7f4cc8154950 2023-12-13T23:55:50.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43310/0 (socket says 172.21.15.124:43310) 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 -- 172.21.15.124:0/2069659002 learned_addr learned my addr 172.21.15.124:0/2069659002 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 0x7f4cc81467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc6d9d640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8144540 0x7f4cc81449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 -- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 msgr2=0x7f4cc81467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 0x7f4cc81467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 -- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8144540 msgr2=0x7f4cc81449d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8144540 0x7f4cc81449d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 -- 172.21.15.124:0/2069659002 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cc8155130 con 0x7f4cb00047c0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 0x7f4cc81467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cc659c640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800fb10 tx=0x7f4cb8002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8006d60 con 0x7f4cb00047c0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb8003970 con 0x7f4cb00047c0 2023-12-13T23:55:50.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cc8155410 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.769+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb8003d20 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.770+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cc80af240 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.771+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cc8155c60 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.771+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4cb800d860 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.771+0000 7f4cb6ffd640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 0x7f4ca806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.772+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cb8089450 con 0x7f4cb00047c0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.772+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 0x7f4ca806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.772+0000 7f4cc5d9b640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 0x7f4ca806d680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc034000 tx=0x7f4cbc007000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:50.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.776+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4cc8155c60 con 0x7f4cb00047c0 2023-12-13T23:55:50.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.944+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4cc8155c60 con 0x7f4cb00047c0 2023-12-13T23:55:50.943 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.945+0000 7f4cb6ffd640 1 -- 172.21.15.124:0/2069659002 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4cc8155c60 con 0x7f4cb00047c0 2023-12-13T23:55:50.943 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:50.945 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 msgr2=0x7f4ca806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 0x7f4ca806d680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc034000 tx=0x7f4cbc007000 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 msgr2=0x7f4cc80aed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4cb800fb10 tx=0x7f4cb8002170 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 shutdown_connections 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4ca806b1c0 0x7f4ca806d680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4cc8144540 0x7f4cc81449d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4cc8154950 0x7f4cc81467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 --2- 172.21.15.124:0/2069659002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4cb00047c0 0x7f4cc80aed00 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 >> 172.21.15.124:0/2069659002 conn(0x7f4cc80b1e50 msgr2=0x7f4cc80b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 shutdown_connections 2023-12-13T23:55:50.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:50.947+0000 7f4cccb5a640 1 -- 172.21.15.124:0/2069659002 wait complete. 2023-12-13T23:55:50.956 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:51.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.060+0000 7f6674661640 1 Processor -- start 2023-12-13T23:55:51.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.060+0000 7f6674661640 1 -- start start 2023-12-13T23:55:51.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.061+0000 7f6674661640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f666c15bbe0 0x7f666c0cebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.061+0000 7f6674661640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f666c15f280 0x7f666c15f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.061+0000 7f6674661640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f666c0cf0f0 con 0x7f666c15bfc0 2023-12-13T23:55:51.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.061+0000 7f6674661640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f666c0c7ee0 con 0x7f666c15f280 2023-12-13T23:55:51.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.061+0000 7f6674661640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f666c0cf330 con 0x7f666c15bbe0 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f666c15f280 0x7f666c15f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f666c15f280 0x7f666c15f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43322/0 (socket says 172.21.15.124:43322) 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 -- 172.21.15.124:0/984951984 learned_addr learned my addr 172.21.15.124:0/984951984 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f66723fd640 1 --2- 172.21.15.124:0/984951984 >> v2:172.21.15.203:3300/0 conn(0x7f666c15bbe0 0x7f666c0cebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 -- 172.21.15.124:0/984951984 >> v2:172.21.15.203:3300/0 conn(0x7f666c15bbe0 msgr2=0x7f666c0cebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 --2- 172.21.15.124:0/984951984 >> v2:172.21.15.203:3300/0 conn(0x7f666c15bbe0 0x7f666c0cebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 -- 172.21.15.124:0/984951984 >> v1:172.21.15.203:6789/0 conn(0x7f666c15bfc0 legacy=0x7f666c0c5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 -- 172.21.15.124:0/984951984 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c0c7920 con 0x7f666c15f280 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.062+0000 7f6672bfe640 1 --2- 172.21.15.124:0/984951984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15f280 0x7f666c15f6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f666c0741e0 tx=0x7f666402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b8c833dddf573 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f66713fb640 1 -- 172.21.15.124:0/984951984 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66640317e0 con 0x7f666c15f280 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f66713fb640 1 -- 172.21.15.124:0/984951984 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6664031940 con 0x7f666c15f280 2023-12-13T23:55:51.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f66713fb640 1 -- 172.21.15.124:0/984951984 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664031cd0 con 0x7f666c15f280 2023-12-13T23:55:51.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 -- 172.21.15.124:0/984951984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15f280 msgr2=0x7f666c15f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 --2- 172.21.15.124:0/984951984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15f280 0x7f666c15f6d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f666c0741e0 tx=0x7f666402b880 comp rx=0 tx=0).stop 2023-12-13T23:55:51.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 -- 172.21.15.124:0/984951984 shutdown_connections 2023-12-13T23:55:51.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 --2- 172.21.15.124:0/984951984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15f280 0x7f666c15f6d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 --2- 172.21.15.124:0/984951984 >> v2:172.21.15.203:3300/0 conn(0x7f666c15bbe0 0x7f666c0cebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 -- 172.21.15.124:0/984951984 >> 172.21.15.124:0/984951984 conn(0x7f666c0b2c10 msgr2=0x7f666c0b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 -- 172.21.15.124:0/984951984 shutdown_connections 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.063+0000 7f6674661640 1 -- 172.21.15.124:0/984951984 wait complete. 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.064+0000 7f6674661640 1 Processor -- start 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.064+0000 7f6674661640 1 -- start start 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f666c15bbe0 0x7f666c157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f666c15f280 0x7f666c150d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c0c88e0 con 0x7f666c15bfc0 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c15e840 con 0x7f666c15bbe0 2023-12-13T23:55:51.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6674661640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f666c15caa0 con 0x7f666c15f280 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6671bfc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6671bfc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43326/0 (socket says 172.21.15.124:43326) 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6671bfc640 1 -- 172.21.15.124:0/3138229297 learned_addr learned my addr 172.21.15.124:0/3138229297 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f66723fd640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f666c15bbe0 0x7f666c157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.065+0000 7f6672bfe640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f666c15f280 0x7f666c150d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 -- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f666c15f280 msgr2=0x7f666c150d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f666c15f280 0x7f666c150d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 -- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f666c15bbe0 msgr2=0x7f666c157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f666c15bbe0 0x7f666c157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 -- 172.21.15.124:0/3138229297 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640094d0 con 0x7f666c15bfc0 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6671bfc640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f665c00c460 tx=0x7f665c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c012830 con 0x7f666c15bfc0 2023-12-13T23:55:51.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c0129d0 con 0x7f666c15bfc0 2023-12-13T23:55:51.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666c0c6710 con 0x7f666c15bfc0 2023-12-13T23:55:51.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c012d80 con 0x7f666c15bfc0 2023-12-13T23:55:51.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.066+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666c15d060 con 0x7f666c15bfc0 2023-12-13T23:55:51.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.067+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666c069400 con 0x7f666c15bfc0 2023-12-13T23:55:51.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.067+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f665c01a3b0 con 0x7f666c15bfc0 2023-12-13T23:55:51.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.068+0000 7f6662ffd640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 0x7f665006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.068+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f665c0895d0 con 0x7f666c15bfc0 2023-12-13T23:55:51.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.068+0000 7f66723fd640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 0x7f665006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.068+0000 7f66723fd640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 0x7f665006d6b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6668009a60 tx=0x7f6668003990 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.072+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f665c054390 con 0x7f666c15bfc0 2023-12-13T23:55:51.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.239+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f666c0c60d0 con 0x7f666c15bfc0 2023-12-13T23:55:51.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.239+0000 7f6662ffd640 1 -- 172.21.15.124:0/3138229297 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f665c0579b0 con 0x7f666c15bfc0 2023-12-13T23:55:51.238 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 msgr2=0x7f665006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 0x7f665006d6b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f6668009a60 tx=0x7f6668003990 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 msgr2=0x7f666c15c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f665c00c460 tx=0x7f665c00c920 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 shutdown_connections 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f665006b1f0 0x7f665006d6b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f666c15f280 0x7f666c150d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f666c15bfc0 0x7f666c15c450 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 --2- 172.21.15.124:0/3138229297 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f666c15bbe0 0x7f666c157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 >> 172.21.15.124:0/3138229297 conn(0x7f666c0b2c10 msgr2=0x7f666c1550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:51.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 shutdown_connections 2023-12-13T23:55:51.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.241+0000 7f6674661640 1 -- 172.21.15.124:0/3138229297 wait complete. 2023-12-13T23:55:51.251 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:51.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.352+0000 7f66a14e7640 1 Processor -- start 2023-12-13T23:55:51.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.352+0000 7f66a14e7640 1 -- start start 2023-12-13T23:55:51.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f66a14e7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f669c15d6c0 0x7f669c06b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f66a14e7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c06ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f66a14e7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669c06ce00 con 0x7f669c15d2e0 2023-12-13T23:55:51.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f66a14e7640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669c06bf40 con 0x7f669c15d6c0 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f66a14e7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669c06c1f0 con 0x7f669c0d1d50 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f669a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f669c15d6c0 0x7f669c06b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c06ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f669a7fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f669c15d6c0 0x7f669c06b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:42820/0 (socket says 172.21.15.124:42820) 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f669a7fc640 1 -- 172.21.15.124:0/3164476802 learned_addr learned my addr 172.21.15.124:0/3164476802 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.353+0000 7f669affd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f669c15d2e0 0x7f669c0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:47936/0 (socket says 172.21.15.124:47936) 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.354+0000 7f669a7fc640 1 -- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 msgr2=0x7f669c06ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.354+0000 7f669a7fc640 1 --2- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c06ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.354+0000 7f669a7fc640 1 -- 172.21.15.124:0/3164476802 >> v1:172.21.15.203:6789/0 conn(0x7f669c15d2e0 legacy=0x7f669c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.354+0000 7f669a7fc640 1 -- 172.21.15.124:0/3164476802 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c06c4d0 con 0x7f669c15d6c0 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f669a7fc640 1 --2- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d6c0 0x7f669c06b4c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6690009870 tx=0x7f669002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c0056872462732c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f6699ffb640 1 -- 172.21.15.124:0/3164476802 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690031640 con 0x7f669c15d6c0 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f6699ffb640 1 -- 172.21.15.124:0/3164476802 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66900317a0 con 0x7f669c15d6c0 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f6699ffb640 1 -- 172.21.15.124:0/3164476802 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6690031b30 con 0x7f669c15d6c0 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 -- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d6c0 msgr2=0x7f669c06b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 --2- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d6c0 0x7f669c06b4c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6690009870 tx=0x7f669002b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 -- 172.21.15.124:0/3164476802 shutdown_connections 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 --2- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c06ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 --2- 172.21.15.124:0/3164476802 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d6c0 0x7f669c06b4c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 -- 172.21.15.124:0/3164476802 >> 172.21.15.124:0/3164476802 conn(0x7f669c09c580 msgr2=0x7f669c0d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 -- 172.21.15.124:0/3164476802 shutdown_connections 2023-12-13T23:55:51.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.355+0000 7f66a14e7640 1 -- 172.21.15.124:0/3164476802 wait complete. 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 Processor -- start 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 -- start start 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c0b7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d2e0 0x7f669c0ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c14a0d0 con 0x7f669c15d6c0 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f669c14c5b0 con 0x7f669c15d2e0 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.357+0000 7f66a14e7640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669c14c890 con 0x7f669c0d1d50 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43338/0 (socket says 172.21.15.124:43338) 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 -- 172.21.15.124:0/2764031369 learned_addr learned my addr 172.21.15.124:0/2764031369 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669affd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c0b7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669a7fc640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d2e0 0x7f669c0ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 -- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 msgr2=0x7f669c0b7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c0b7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 -- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d2e0 msgr2=0x7f669c0ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d2e0 0x7f669c0ccd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 -- 172.21.15.124:0/2764031369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66900094d0 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f669b7fe640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f669400e930 tx=0x7f669400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6694009690 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f669400c980 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669c14cb70 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f669400cd30 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.358+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669c14d130 con 0x7f669c15d6c0 2023-12-13T23:55:51.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.359+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669c14d320 con 0x7f669c15d6c0 2023-12-13T23:55:51.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.360+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6694013470 con 0x7f669c15d6c0 2023-12-13T23:55:51.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.360+0000 7f66837fe640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 0x7f668806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.360+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6694089b80 con 0x7f669c15d6c0 2023-12-13T23:55:51.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.360+0000 7f669affd640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 0x7f668806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.361+0000 7f669affd640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 0x7f668806d680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f668c0012e0 tx=0x7f668c0016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.364+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6694054a20 con 0x7f669c15d6c0 2023-12-13T23:55:51.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.541+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f669c067420 con 0x7f669c15d6c0 2023-12-13T23:55:51.540 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.542+0000 7f66837fe640 1 -- 172.21.15.124:0/2764031369 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6694017080 con 0x7f669c15d6c0 2023-12-13T23:55:51.540 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 msgr2=0x7f668806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 0x7f668806d680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f668c0012e0 tx=0x7f668c0016f0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 msgr2=0x7f669c0b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f669400e930 tx=0x7f669400edf0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 shutdown_connections 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f668806b1c0 0x7f668806d680 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f669c15d2e0 0x7f669c0ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f669c15d6c0 0x7f669c0b0700 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 --2- 172.21.15.124:0/2764031369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f669c0d1d50 0x7f669c0b7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 >> 172.21.15.124:0/2764031369 conn(0x7f669c09c580 msgr2=0x7f669c15c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:51.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 shutdown_connections 2023-12-13T23:55:51.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.544+0000 7f66a14e7640 1 -- 172.21.15.124:0/2764031369 wait complete. 2023-12-13T23:55:51.553 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi203.front.sepia.ceph.com... 2023-12-13T23:55:51.553 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:51.554 DEBUG:teuthology.orchestra.run.smithi203:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2023-12-13T23:55:51.587 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:55:51.587 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:55:51.587 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:51.587 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-13T23:55:51.770 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:51.770 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:51.785 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:51.785 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:51.841 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:51.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.981+0000 7f6d7c44f640 1 Processor -- start 2023-12-13T23:55:51.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.981+0000 7f6d7c44f640 1 -- start start 2023-12-13T23:55:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7c44f640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6d7415d330 0x7f6d740b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7c44f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6d740d2910 0x7f6d740d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7c44f640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d740b2b10 con 0x7f6d740d3730 2023-12-13T23:55:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7c44f640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d740d3bc0 con 0x7f6d7415d330 2023-12-13T23:55:51.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7c44f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d740d1590 con 0x7f6d740d2910 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7a1eb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6d7415d330 0x7f6d740b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7a9ec640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6d740d3730 0x7f6d740b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40780/0 (socket says 172.21.15.124:40780) 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.982+0000 7f6d7a9ec640 1 -- 172.21.15.124:0/2000913899 learned_addr learned my addr 172.21.15.124:0/2000913899 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d799ea640 1 --2- 172.21.15.124:0/2000913899 >> v2:172.21.15.203:3300/0 conn(0x7f6d740d2910 0x7f6d740d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d7a1eb640 1 -- 172.21.15.124:0/2000913899 >> v2:172.21.15.203:3300/0 conn(0x7f6d740d2910 msgr2=0x7f6d740d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d7a1eb640 1 --2- 172.21.15.124:0/2000913899 >> v2:172.21.15.203:3300/0 conn(0x7f6d740d2910 0x7f6d740d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d7a1eb640 1 -- 172.21.15.124:0/2000913899 >> v1:172.21.15.124:6789/0 conn(0x7f6d740d3730 legacy=0x7f6d740b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d791e9640 1 -- 172.21.15.124:0/2000913899 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445975941 0 0) 0x7f6d740b2b10 con 0x7f6d740d3730 2023-12-13T23:55:51.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d7a1eb640 1 -- 172.21.15.124:0/2000913899 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d740d1870 con 0x7f6d7415d330 2023-12-13T23:55:51.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.983+0000 7f6d7a1eb640 1 --2- 172.21.15.124:0/2000913899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d740b8b80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6d740d1540 tx=0x7f6d6c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b339f83078ce4e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d791e9640 1 -- 172.21.15.124:0/2000913899 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c02cbf0 con 0x7f6d7415d330 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d791e9640 1 -- 172.21.15.124:0/2000913899 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c02cd50 con 0x7f6d7415d330 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d791e9640 1 -- 172.21.15.124:0/2000913899 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6c034660 con 0x7f6d7415d330 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2000913899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 msgr2=0x7f6d740b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2000913899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d740b8b80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6d740d1540 tx=0x7f6d6c02be40 comp rx=0 tx=0).stop 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2000913899 shutdown_connections 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2000913899 >> v2:172.21.15.203:3300/0 conn(0x7f6d740d2910 0x7f6d740d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2000913899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d740b8b80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2000913899 >> 172.21.15.124:0/2000913899 conn(0x7f6d74072020 msgr2=0x7f6d74072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2000913899 shutdown_connections 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.984+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2000913899 wait complete. 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.985+0000 7f6d7c44f640 1 Processor -- start 2023-12-13T23:55:51.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.985+0000 7f6d7c44f640 1 -- start start 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740d2910 0x7f6d74149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d740d3730 0x7f6d7415f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d740cb4b0 con 0x7f6d7415d330 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d74149870 con 0x7f6d740d2910 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7c44f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d74160040 con 0x7f6d740d3730 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7a9ec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7a9ec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43348/0 (socket says 172.21.15.124:43348) 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7a9ec640 1 -- 172.21.15.124:0/2204943077 learned_addr learned my addr 172.21.15.124:0/2204943077 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7a9ec640 1 -- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d740d3730 msgr2=0x7f6d7415f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.986+0000 7f6d7a9ec640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d740d3730 0x7f6d7415f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7a9ec640 1 -- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740d2910 msgr2=0x7f6d74149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:51.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7a9ec640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740d2910 0x7f6d74149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7a9ec640 1 -- 172.21.15.124:0/2204943077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c009a70 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7a9ec640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000c450 tx=0x7f6d7000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d700074e0 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d70007660 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70004a20 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d74160320 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.987+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d741608e0 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.988+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d74069400 con 0x7f6d7415d330 2023-12-13T23:55:51.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.988+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6d70004b80 con 0x7f6d7415d330 2023-12-13T23:55:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.989+0000 7f6d6affd640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 0x7f6d5806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.989+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d70088ae0 con 0x7f6d7415d330 2023-12-13T23:55:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.989+0000 7f6d7a1eb640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 0x7f6d5806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:51.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.990+0000 7f6d7a1eb640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 0x7f6d5806d680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c036000 tx=0x7f6d6c02e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:51.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:51.993+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6d70053960 con 0x7f6d7415d330 2023-12-13T23:55:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.163+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d740cb5f0 con 0x7f6d7415d330 2023-12-13T23:55:52.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.164+0000 7f6d6affd640 1 -- 172.21.15.124:0/2204943077 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16566 (secure 0 0 0) 0x7f6d70018020 con 0x7f6d7415d330 2023-12-13T23:55:52.163 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:52.163 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":21,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:13.733994+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 msgr2=0x7f6d5806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 0x7f6d5806d680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f6d6c036000 tx=0x7f6d6c02e000 comp rx=0 tx=0).stop 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 msgr2=0x7f6d7415d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6d7000c450 tx=0x7f6d7000c910 comp rx=0 tx=0).stop 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 shutdown_connections 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6d5806b1c0 0x7f6d5806d680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6d7415d330 0x7f6d7415d710 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6d740d3730 0x7f6d7415f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 --2- 172.21.15.124:0/2204943077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6d740d2910 0x7f6d74149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 >> 172.21.15.124:0/2204943077 conn(0x7f6d74072020 msgr2=0x7f6d74072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 shutdown_connections 2023-12-13T23:55:52.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.166+0000 7f6d7c44f640 1 -- 172.21.15.124:0/2204943077 wait complete. 2023-12-13T23:55:52.176 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:52.176 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:52.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.281+0000 7ff07ee9b640 1 Processor -- start 2023-12-13T23:55:52.280 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.281+0000 7ff07ee9b640 1 -- start start 2023-12-13T23:55:52.281 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07ee9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff078143490 0x7ff0780b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.281 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07ee9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff07815e730 0x7ff0780b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.281 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07ee9b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff07815e2a0 con 0x7ff07815a4e0 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07ee9b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff07815b520 con 0x7ff07815e730 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07ee9b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff078143870 con 0x7ff078143490 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff07815e730 0x7ff0780b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff07815e730 0x7ff0780b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43358/0 (socket says 172.21.15.124:43358) 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 -- 172.21.15.124:0/334970235 learned_addr learned my addr 172.21.15.124:0/334970235 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff07cc37640 1 --2- 172.21.15.124:0/334970235 >> v2:172.21.15.203:3300/0 conn(0x7ff078143490 0x7ff0780b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 -- 172.21.15.124:0/334970235 >> v2:172.21.15.203:3300/0 conn(0x7ff078143490 msgr2=0x7ff0780b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 --2- 172.21.15.124:0/334970235 >> v2:172.21.15.203:3300/0 conn(0x7ff078143490 0x7ff0780b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 -- 172.21.15.124:0/334970235 >> v1:172.21.15.203:6789/0 conn(0x7ff07815a4e0 legacy=0x7ff07815a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.282+0000 7ff077fff640 1 -- 172.21.15.124:0/334970235 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff07815b240 con 0x7ff07815e730 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff077fff640 1 --2- 172.21.15.124:0/334970235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815e730 0x7ff0780b1230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff07806e620 tx=0x7ff06402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6b6c7bd99225db server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff0777fe640 1 -- 172.21.15.124:0/334970235 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064031870 con 0x7ff07815e730 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff0777fe640 1 -- 172.21.15.124:0/334970235 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0640319d0 con 0x7ff07815e730 2023-12-13T23:55:52.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff0777fe640 1 -- 172.21.15.124:0/334970235 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064031d60 con 0x7ff07815e730 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 -- 172.21.15.124:0/334970235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815e730 msgr2=0x7ff0780b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/334970235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815e730 0x7ff0780b1230 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7ff07806e620 tx=0x7ff06402b910 comp rx=0 tx=0).stop 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 -- 172.21.15.124:0/334970235 shutdown_connections 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/334970235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815e730 0x7ff0780b1230 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/334970235 >> v2:172.21.15.203:3300/0 conn(0x7ff078143490 0x7ff0780b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.283+0000 7ff07ee9b640 1 -- 172.21.15.124:0/334970235 >> 172.21.15.124:0/334970235 conn(0x7ff07806a4d0 msgr2=0x7ff07806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.285 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.284+0000 7ff07ee9b640 1 -- 172.21.15.124:0/334970235 shutdown_connections 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.284+0000 7ff07ee9b640 1 -- 172.21.15.124:0/334970235 wait complete. 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 Processor -- start 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 -- start start 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff078143490 0x7ff078140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff07815e730 0x7ff07813e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078143060 con 0x7ff07815a4e0 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.285+0000 7ff07ee9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff078141c00 con 0x7ff078143490 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff07ee9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff078141360 con 0x7ff07815e730 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43362/0 (socket says 172.21.15.124:43362) 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 -- 172.21.15.124:0/3789119983 learned_addr learned my addr 172.21.15.124:0/3789119983 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:52.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff07cc37640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff078143490 0x7ff078140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff07d438640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff07815e730 0x7ff07813e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 -- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff07815e730 msgr2=0x7ff07813e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff07815e730 0x7ff07813e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 -- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff078143490 msgr2=0x7ff078140c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff078143490 0x7ff078140c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 -- 172.21.15.124:0/3789119983 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640094d0 con 0x7ff07815a4e0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff077fff640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff064000c00 tx=0x7ff0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06403abe0 con 0x7ff07815a4e0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff06403ad40 con 0x7ff07815a4e0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff078141640 con 0x7ff07815a4e0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff064044610 con 0x7ff07815a4e0 2023-12-13T23:55:52.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.286+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07813bcd0 con 0x7ff07815a4e0 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.287+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0780b4380 con 0x7ff07815a4e0 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.288+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff06404d450 con 0x7ff07815a4e0 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.288+0000 7ff0757fa640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 0x7ff05c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.288+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff06402d040 con 0x7ff07815a4e0 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.288+0000 7ff07cc37640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 0x7ff05c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.289+0000 7ff07cc37640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 0x7ff05c06d680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.292+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff064076c30 con 0x7ff07815a4e0 2023-12-13T23:55:52.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.459+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff078069400 con 0x7ff07815a4e0 2023-12-13T23:55:52.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.459+0000 7ff0757fa640 1 -- 172.21.15.124:0/3789119983 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff06407a250 con 0x7ff07815a4e0 2023-12-13T23:55:52.458 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 msgr2=0x7ff05c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 0x7ff05c06d680 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7ff06c006fd0 tx=0x7ff06c008040 comp rx=0 tx=0).stop 2023-12-13T23:55:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 msgr2=0x7ff078142b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff064000c00 tx=0x7ff0640046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 shutdown_connections 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff05c06b1c0 0x7ff05c06d680 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff07815e730 0x7ff07813e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff07815a4e0 0x7ff078142b20 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 --2- 172.21.15.124:0/3789119983 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff078143490 0x7ff078140c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.461+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 >> 172.21.15.124:0/3789119983 conn(0x7ff07806a4d0 msgr2=0x7ff07806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.462+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 shutdown_connections 2023-12-13T23:55:52.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.462+0000 7ff07ee9b640 1 -- 172.21.15.124:0/3789119983 wait complete. 2023-12-13T23:55:52.472 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:52.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.575+0000 7f7747a5d640 1 Processor -- start 2023-12-13T23:55:52.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.576+0000 7f7747a5d640 1 -- start start 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.576+0000 7f7747a5d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.576+0000 7f7747a5d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 0x7f774006ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.576+0000 7f7747a5d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774006ce00 con 0x7f774015d2e0 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.576+0000 7f7747a5d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774006bf40 con 0x7f77400d1d50 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7747a5d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774006c1f0 con 0x7f774015d6c0 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7745ffa640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 0x7f774006ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:46476/0 (socket says 172.21.15.124:46476) 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 -- 172.21.15.124:0/709015571 learned_addr learned my addr 172.21.15.124:0/709015571 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 -- 172.21.15.124:0/709015571 >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 msgr2=0x7f774006ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 --2- 172.21.15.124:0/709015571 >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 0x7f774006ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 -- 172.21.15.124:0/709015571 >> v1:172.21.15.203:6789/0 conn(0x7f774015d2e0 legacy=0x7f77400d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7744ff8640 1 -- 172.21.15.124:0/709015571 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774006c4d0 con 0x7f774015d6c0 2023-12-13T23:55:52.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.577+0000 7f7745ffa640 1 --2- 172.21.15.124:0/709015571 >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 0x7f774006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.578+0000 7f7744ff8640 1 --2- 172.21.15.124:0/709015571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7730009870 tx=0x7f773002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=199b386754bba056 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.578+0000 7f7737fff640 1 -- 172.21.15.124:0/709015571 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7730031640 con 0x7f774015d6c0 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.578+0000 7f7737fff640 1 -- 172.21.15.124:0/709015571 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77300317a0 con 0x7f774015d6c0 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.578+0000 7f7737fff640 1 -- 172.21.15.124:0/709015571 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7730031b30 con 0x7f774015d6c0 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 -- 172.21.15.124:0/709015571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 msgr2=0x7f774006b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 --2- 172.21.15.124:0/709015571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7730009870 tx=0x7f773002b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 -- 172.21.15.124:0/709015571 shutdown_connections 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 --2- 172.21.15.124:0/709015571 >> v2:172.21.15.203:3300/0 conn(0x7f77400d1d50 0x7f774006ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 --2- 172.21.15.124:0/709015571 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f774015d6c0 0x7f774006b4c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 -- 172.21.15.124:0/709015571 >> 172.21.15.124:0/709015571 conn(0x7f774009c580 msgr2=0x7f77400d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 -- 172.21.15.124:0/709015571 shutdown_connections 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.579+0000 7f7747a5d640 1 -- 172.21.15.124:0/709015571 wait complete. 2023-12-13T23:55:52.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.580+0000 7f7747a5d640 1 Processor -- start 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.580+0000 7f7747a5d640 1 -- start start 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77400d1d50 0x7f774015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f774015d2e0 0x7f774015f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77401544d0 con 0x7f774015d6c0 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77400cafe0 con 0x7f774015d2e0 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774015f7a0 con 0x7f77400d1d50 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43370/0 (socket says 172.21.15.124:43370) 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 -- 172.21.15.124:0/3966999947 learned_addr learned my addr 172.21.15.124:0/3966999947 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f77457f9640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77400d1d50 0x7f774015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7744ff8640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f774015d2e0 0x7f774015f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 -- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77400d1d50 msgr2=0x7f774015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77400d1d50 0x7f774015ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 -- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f774015d2e0 msgr2=0x7f774015f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f774015d2e0 0x7f774015f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 -- 172.21.15.124:0/3966999947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7738009a70 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7745ffa640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f773800efc0 tx=0x7f7738009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7738009d70 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773800f040 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77300094d0 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.581+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7738003990 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.582+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774015ffc0 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.583+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77400cb150 con 0x7f774015d6c0 2023-12-13T23:55:52.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.583+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f77380144a0 con 0x7f774015d6c0 2023-12-13T23:55:52.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.583+0000 7f7735ffb640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 0x7f772406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.584+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f773808a6a0 con 0x7f774015d6c0 2023-12-13T23:55:52.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.584+0000 7f77457f9640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 0x7f772406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.584+0000 7f77457f9640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 0x7f772406d680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77400ca830 tx=0x7f773c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.588+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f77380553d0 con 0x7f774015d6c0 2023-12-13T23:55:52.755 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.756+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7740067420 con 0x7f774015d6c0 2023-12-13T23:55:52.755 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.757+0000 7f7735ffb640 1 -- 172.21.15.124:0/3966999947 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+11 (secure 0 0 0) 0x7f77380589f0 con 0x7f774015d6c0 2023-12-13T23:55:52.756 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 msgr2=0x7f772406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 0x7f772406d680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f77400ca830 tx=0x7f773c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 msgr2=0x7f77401549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f773800efc0 tx=0x7f7738009f90 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 shutdown_connections 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f772406b1c0 0x7f772406d680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f774015d2e0 0x7f774015f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f774015d6c0 0x7f77401549d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 --2- 172.21.15.124:0/3966999947 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f77400d1d50 0x7f774015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 >> 172.21.15.124:0/3966999947 conn(0x7f774009c580 msgr2=0x7f77400b44d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 shutdown_connections 2023-12-13T23:55:52.759 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.759+0000 7f7747a5d640 1 -- 172.21.15.124:0/3966999947 wait complete. 2023-12-13T23:55:52.768 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:52.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.873+0000 7fd9d241c640 1 Processor -- start 2023-12-13T23:55:52.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.873+0000 7fd9d241c640 1 -- start start 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d241c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 0x7fd9cc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d241c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc0b2b10 con 0x7fd9cc0d2d90 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d241c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9cc0d1590 con 0x7fd9cc0d2910 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d241c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0d1870 con 0x7fd9cc15d330 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d09b9640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd9cc0d2d90 0x7fd9cc0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40782/0 (socket says 172.21.15.124:40782) 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9d09b9640 1 -- 172.21.15.124:0/3902063529 learned_addr learned my addr 172.21.15.124:0/3902063529 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9cbfff640 1 --2- 172.21.15.124:0/3902063529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 0x7fd9cc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9cbfff640 1 -- 172.21.15.124:0/3902063529 >> v1:172.21.15.203:6789/0 conn(0x7fd9cc0d2910 legacy=0x7fd9cc0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9cbfff640 1 -- 172.21.15.124:0/3902063529 >> v1:172.21.15.124:6789/0 conn(0x7fd9cc0d2d90 legacy=0x7fd9cc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.874+0000 7fd9cbfff640 1 -- 172.21.15.124:0/3902063529 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc0d1b50 con 0x7fd9cc15d330 2023-12-13T23:55:52.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.875+0000 7fd9cbfff640 1 --2- 172.21.15.124:0/3902063529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 0x7fd9cc0b8b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097d0 tx=0x7fd9bc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=646eeda74baed689 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.876+0000 7fd9caffd640 1 -- 172.21.15.124:0/3902063529 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc031800 con 0x7fd9cc15d330 2023-12-13T23:55:52.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.876+0000 7fd9caffd640 1 -- 172.21.15.124:0/3902063529 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc031960 con 0x7fd9cc15d330 2023-12-13T23:55:52.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.876+0000 7fd9caffd640 1 -- 172.21.15.124:0/3902063529 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc031cf0 con 0x7fd9cc15d330 2023-12-13T23:55:52.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 -- 172.21.15.124:0/3902063529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 msgr2=0x7fd9cc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 --2- 172.21.15.124:0/3902063529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 0x7fd9cc0b8b80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc0097d0 tx=0x7fd9bc02b8a0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 -- 172.21.15.124:0/3902063529 shutdown_connections 2023-12-13T23:55:52.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 --2- 172.21.15.124:0/3902063529 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc15d330 0x7fd9cc0b8b80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.876 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 -- 172.21.15.124:0/3902063529 >> 172.21.15.124:0/3902063529 conn(0x7fd9cc072020 msgr2=0x7fd9cc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:52.876 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 -- 172.21.15.124:0/3902063529 shutdown_connections 2023-12-13T23:55:52.876 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.877+0000 7fd9d241c640 1 -- 172.21.15.124:0/3902063529 wait complete. 2023-12-13T23:55:52.876 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.878+0000 7fd9d241c640 1 Processor -- start 2023-12-13T23:55:52.876 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.878+0000 7fd9d241c640 1 -- start start 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc0d2910 0x7fd9cc152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9cc0d2d90 0x7fd9cc0c7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0c8590 con 0x7fd9cc15d330 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0c8870 con 0x7fd9cc0d2d90 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d241c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9cc0c8b50 con 0x7fd9cc0d2910 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43386/0 (socket says 172.21.15.124:43386) 2023-12-13T23:55:52.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 -- 172.21.15.124:0/2218591325 learned_addr learned my addr 172.21.15.124:0/2218591325 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9cbfff640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc0d2910 0x7fd9cc152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9cb7fe640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9cc0d2d90 0x7fd9cc0c7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 -- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc0d2910 msgr2=0x7fd9cc152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc0d2910 0x7fd9cc152760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 -- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9cc0d2d90 msgr2=0x7fd9cc0c7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9cc0d2d90 0x7fd9cc0c7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.879+0000 7fd9d09b9640 1 -- 172.21.15.124:0/2218591325 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc009400 con 0x7fd9cc15d330 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9d09b9640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400caf0 tx=0x7fd9c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c400aca0 con 0x7fd9cc15d330 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c400ae00 con 0x7fd9cc15d330 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4012600 con 0x7fd9cc15d330 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9cc0c8e30 con 0x7fd9cc15d330 2023-12-13T23:55:52.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.880+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9cc152ca0 con 0x7fd9cc15d330 2023-12-13T23:55:52.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.881+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9cc0c9350 con 0x7fd9cc15d330 2023-12-13T23:55:52.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.882+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd9c4007070 con 0x7fd9cc15d330 2023-12-13T23:55:52.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.882+0000 7fd9c8ff9640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 0x7fd9b806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:52.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.882+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd9c40898d0 con 0x7fd9cc15d330 2023-12-13T23:55:52.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.882+0000 7fd9cbfff640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 0x7fd9b806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:52.882 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.883+0000 7fd9cbfff640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 0x7fd9b806d680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc036000 tx=0x7fd9bc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:52.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:52.888+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd9cc0c9350 con 0x7fd9cc15d330 2023-12-13T23:55:53.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.055+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd9cc0c9350 con 0x7fd9cc15d330 2023-12-13T23:55:53.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.055+0000 7fd9c8ff9640 1 -- 172.21.15.124:0/2218591325 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd9cc0c9350 con 0x7fd9cc15d330 2023-12-13T23:55:53.054 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 msgr2=0x7fd9b806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 0x7fd9b806d680 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc036000 tx=0x7fd9bc03c040 comp rx=0 tx=0).stop 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 msgr2=0x7fd9cc0c7f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400caf0 tx=0x7fd9c400a060 comp rx=0 tx=0).stop 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 shutdown_connections 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd9b806b1c0 0x7fd9b806d680 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd9cc0d2d90 0x7fd9cc0c7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd9cc15d330 0x7fd9cc0c7f60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 --2- 172.21.15.124:0/2218591325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd9cc0d2910 0x7fd9cc152760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 >> 172.21.15.124:0/2218591325 conn(0x7fd9cc072020 msgr2=0x7fd9cc0b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 shutdown_connections 2023-12-13T23:55:53.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.057+0000 7fd9d241c640 1 -- 172.21.15.124:0/2218591325 wait complete. 2023-12-13T23:55:53.067 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:53.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.169+0000 7fe3c1ccf640 1 Processor -- start 2023-12-13T23:55:53.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.169+0000 7fe3c1ccf640 1 -- start start 2023-12-13T23:55:53.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3c1ccf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3bc0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3c1ccf640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc071770 con 0x7fe3bc144980 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3c1ccf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3bc0ccb60 con 0x7fe3bc0ceb70 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3c1ccf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc15ced0 con 0x7fe3bc0b4370 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3bb7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe3bc144980 0x7fe3bc15c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40790/0 (socket says 172.21.15.124:40790) 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3bb7fe640 1 -- 172.21.15.124:0/73441900 learned_addr learned my addr 172.21.15.124:0/73441900 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901506850 0 0) 0x7fe3bc071770 con 0x7fe3bc144980 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3bc0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.170+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a4003520 con 0x7fe3bc144980 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1771763250 0 0) 0x7fe3a4003520 con 0x7fe3bc144980 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 msgr2=0x7fe3bc0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3bc0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 >> v1:172.21.15.203:6789/0 conn(0x7fe3bc0ceb70 legacy=0x7fe3bc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0b4ce0 con 0x7fe3bc144980 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fe3ac001670 con 0x7fe3bc144980 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 >> v1:172.21.15.124:6789/0 conn(0x7fe3bc144980 legacy=0x7fe3bc15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a40039a0 0x7fe3a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a4004d00 0x7fe3a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc15ced0 con 0x7fe3a4004d00 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a400ba30 con 0x7fe3a40039a0 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a400bb70 con 0x7fe3bc0b4370 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a4004d00 0x7fe3a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bb7fe640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a40039a0 0x7fe3a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3baffd640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 -- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 msgr2=0x7fe3a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 -- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a40039a0 msgr2=0x7fe3a4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a40039a0 0x7fe3a4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3bbfff640 1 -- 172.21.15.124:0/73441900 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4003520 con 0x7fe3a4004d00 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.171+0000 7fe3baffd640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0b4370 0x7fe3a40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a4004d00 0x7fe3a4005180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fc0 tx=0x7fe3a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd0ecf07da5574b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b402cc20 con 0x7fe3a4004d00 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b400f040 con 0x7fe3a4004d00 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3ba7fc640 1 -- 172.21.15.124:0/73441900 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b4034440 con 0x7fe3a4004d00 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/73441900 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a4004d00 msgr2=0x7fe3a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/73441900 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3a4004d00 0x7fe3a4005180 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009fc0 tx=0x7fe3a40057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3bbfff640 1 -- 172.21.15.124:0/73441900 reap_dead start 2023-12-13T23:55:53.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/73441900 shutdown_connections 2023-12-13T23:55:53.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.172+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/73441900 >> 172.21.15.124:0/73441900 conn(0x7fe3bc06a4d0 msgr2=0x7fe3bc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.173+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/73441900 shutdown_connections 2023-12-13T23:55:53.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.173+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/73441900 wait complete. 2023-12-13T23:55:53.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.174+0000 7fe3c1ccf640 1 Processor -- start 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.174+0000 7fe3c1ccf640 1 -- start start 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a4004d00 0x7fe3bc0a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0a9ef0 0x7fe3bc1f4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0ab110 con 0x7fe3bc1426a0 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0a9af0 con 0x7fe3a4004d00 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3bc0aa490 con 0x7fe3bc0a9ef0 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43402/0 (socket says 172.21.15.124:43402) 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 -- 172.21.15.124:0/1576443700 learned_addr learned my addr 172.21.15.124:0/1576443700 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3bb7fe640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a4004d00 0x7fe3bc0a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3bbfff640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0a9ef0 0x7fe3bc1f4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 -- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0a9ef0 msgr2=0x7fe3bc1f4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0a9ef0 0x7fe3bc1f4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 -- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a4004d00 msgr2=0x7fe3bc0a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a4004d00 0x7fe3bc0a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 -- 172.21.15.124:0/1576443700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc0aa770 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3baffd640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e7d0 tx=0x7fe3b000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b00050d0 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0005230 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3bc0aaa50 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000c6c0 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.175+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3bc0ab530 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.176+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3bc0ab250 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.177+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe3b000c820 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.177+0000 7fe3a3fff640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 0x7fe3a806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.177+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe3b00897b0 con 0x7fe3bc1426a0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.177+0000 7fe3bb7fe640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 0x7fe3a806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.178+0000 7fe3bb7fe640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 0x7fe3a806d680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0017a0 tx=0x7fe3ac0015d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.181+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe3bc0ab250 con 0x7fe3bc1426a0 2023-12-13T23:55:53.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.357+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe3bc0ab250 con 0x7fe3bc1426a0 2023-12-13T23:55:53.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.357+0000 7fe3a3fff640 1 -- 172.21.15.124:0/1576443700 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe3bc0ab250 con 0x7fe3bc1426a0 2023-12-13T23:55:53.356 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 msgr2=0x7fe3a806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 0x7fe3a806d680 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0017a0 tx=0x7fe3ac0015d0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 msgr2=0x7fe3bc1f3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e7d0 tx=0x7fe3b000ebd0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 shutdown_connections 2023-12-13T23:55:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe3a806b1c0 0x7fe3a806d680 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe3bc0a9ef0 0x7fe3bc1f4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3bc1426a0 0x7fe3bc1f3bd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 --2- 172.21.15.124:0/1576443700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe3a4004d00 0x7fe3bc0a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 >> 172.21.15.124:0/1576443700 conn(0x7fe3bc06a4d0 msgr2=0x7fe3bc0cc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 shutdown_connections 2023-12-13T23:55:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.359+0000 7fe3c1ccf640 1 -- 172.21.15.124:0/1576443700 wait complete. 2023-12-13T23:55:53.369 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:55:53.369 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:53.384 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:53.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.525+0000 7f29a1c09640 1 Processor -- start 2023-12-13T23:55:53.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.525+0000 7f29a1c09640 1 -- start start 2023-12-13T23:55:53.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f29a1c09640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 0x7f299c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f29a1c09640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c15e2a0 con 0x7f299c15e730 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f29a1c09640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c15b240 con 0x7f299c143490 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f29a1c09640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c15b520 con 0x7f299c0b1400 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f299bfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f299c15e730 0x7f299c15aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40806/0 (socket says 172.21.15.124:40806) 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f299bfff640 1 -- 172.21.15.124:0/2056932323 learned_addr learned my addr 172.21.15.124:0/2056932323 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f2992ffd640 1 --2- 172.21.15.124:0/2056932323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 0x7f299c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.526+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015093000 0 0) 0x7f299c15e2a0 con 0x7f299c15e730 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988003520 con 0x7f299c15e730 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277001364 0 0) 0x7f299c15b240 con 0x7f299c143490 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f299c15e2a0 con 0x7f299c143490 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f2992ffd640 1 -- 172.21.15.124:0/2056932323 >> v1:172.21.15.203:6789/0 conn(0x7f299c143490 legacy=0x7f299c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f2992ffd640 1 -- 172.21.15.124:0/2056932323 >> v1:172.21.15.124:6789/0 conn(0x7f299c15e730 legacy=0x7f299c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f2992ffd640 1 -- 172.21.15.124:0/2056932323 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c15dc50 con 0x7f299c0b1400 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.527+0000 7f2992ffd640 1 --2- 172.21.15.124:0/2056932323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 0x7f299c15a4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f29840097d0 tx=0x7f298402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28c646d1bf8e5a2e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984031890 con 0x7f299c0b1400 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29840319f0 con 0x7f299c0b1400 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f299affd640 1 -- 172.21.15.124:0/2056932323 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2984031d60 con 0x7f299c0b1400 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 -- 172.21.15.124:0/2056932323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 msgr2=0x7f299c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2056932323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 0x7f299c15a4e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f29840097d0 tx=0x7f298402b930 comp rx=0 tx=0).stop 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 -- 172.21.15.124:0/2056932323 shutdown_connections 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2056932323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c0b1400 0x7f299c15a4e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 -- 172.21.15.124:0/2056932323 >> 172.21.15.124:0/2056932323 conn(0x7f299c06a4d0 msgr2=0x7f299c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 -- 172.21.15.124:0/2056932323 shutdown_connections 2023-12-13T23:55:53.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.528+0000 7f29a1c09640 1 -- 172.21.15.124:0/2056932323 wait complete. 2023-12-13T23:55:53.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.529+0000 7f29a1c09640 1 Processor -- start 2023-12-13T23:55:53.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 -- start start 2023-12-13T23:55:53.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 0x7f299c13f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c143490 0x7f299c0a8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0a9980 con 0x7f299c15e730 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0a9c60 con 0x7f299c0b1400 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f29a1c09640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0a9f40 con 0x7f299c143490 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f299bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f299bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43408/0 (socket says 172.21.15.124:43408) 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f299bfff640 1 -- 172.21.15.124:0/2453677268 learned_addr learned my addr 172.21.15.124:0/2453677268 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f2992ffd640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c143490 0x7f299c0a8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.530+0000 7f299b7fe640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 0x7f299c13f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 -- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c143490 msgr2=0x7f299c0a8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c143490 0x7f299c0a8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 -- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 msgr2=0x7f299c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 0x7f299c13f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 -- 172.21.15.124:0/2453677268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2984009400 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299b7fe640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 0x7f299c13f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f299bfff640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f299400caf0 tx=0x7f299400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299400aca0 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299400ae00 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c0aa220 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994012620 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.531+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c0b0540 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.532+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c143ba0 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.532+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2994007070 con 0x7f299c15e730 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.533+0000 7f2998ff9640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 0x7f297c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.533+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2994089920 con 0x7f299c15e730 2023-12-13T23:55:53.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.533+0000 7f299b7fe640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 0x7f297c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.533+0000 7f299b7fe640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 0x7f297c06d6b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f298c001710 tx=0x7f298c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.537+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2994054710 con 0x7f299c15e730 2023-12-13T23:55:53.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.703+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f299c069400 con 0x7f299c15e730 2023-12-13T23:55:53.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.704+0000 7f2998ff9640 1 -- 172.21.15.124:0/2453677268 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16815 (secure 0 0 0) 0x7f2994017020 con 0x7f299c15e730 2023-12-13T23:55:53.703 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:53.703 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:52.872498+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:53.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 msgr2=0x7f297c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 0x7f297c06d6b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f298c001710 tx=0x7f298c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 msgr2=0x7f299c1595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f299400caf0 tx=0x7f299400a060 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 shutdown_connections 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f297c06b1f0 0x7f297c06d6b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f299c15e730 0x7f299c1595c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f299c143490 0x7f299c0a8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 --2- 172.21.15.124:0/2453677268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f299c0b1400 0x7f299c13f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 >> 172.21.15.124:0/2453677268 conn(0x7f299c06a4d0 msgr2=0x7f299c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 shutdown_connections 2023-12-13T23:55:53.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.706+0000 7f29a1c09640 1 -- 172.21.15.124:0/2453677268 wait complete. 2023-12-13T23:55:53.716 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:53.716 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:53.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.819+0000 7f07b8ffb640 1 Processor -- start 2023-12-13T23:55:53.821 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b8ffb640 1 -- start start 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b8ffb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07b415d330 0x7f07b40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b8ffb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b40b2b10 con 0x7f07b40d3900 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b8ffb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b40d1590 con 0x7f07b40d2910 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b8ffb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b40d1870 con 0x7f07b415d330 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b2d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f07b40d3900 0x7f07b40b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40814/0 (socket says 172.21.15.124:40814) 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.820+0000 7f07b2d9d640 1 -- 172.21.15.124:0/2025519375 learned_addr learned my addr 172.21.15.124:0/2025519375 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b259c640 1 --2- 172.21.15.124:0/2025519375 >> v2:172.21.15.203:3300/0 conn(0x7f07b415d330 0x7f07b40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892702623 0 0) 0x7f07b40b2b10 con 0x7f07b40d3900 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07a0003520 con 0x7f07b40d3900 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237129288 0 0) 0x7f07a0003520 con 0x7f07b40d3900 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 >> v2:172.21.15.203:3300/0 conn(0x7f07b415d330 msgr2=0x7f07b40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 --2- 172.21.15.124:0/2025519375 >> v2:172.21.15.203:3300/0 conn(0x7f07b415d330 0x7f07b40b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.822 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 >> v1:172.21.15.203:6789/0 conn(0x7f07b40d2910 legacy=0x7f07b40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40d1b50 con 0x7f07b40d3900 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f07ac0019f0 con 0x7f07b40d3900 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 >> v1:172.21.15.124:6789/0 conn(0x7f07b40d3900 legacy=0x7f07b40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07a0003970 0x7f07a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 0x7f07a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a0005180 0x7f07a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40d1870 con 0x7f07a00047c0 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a000c120 con 0x7f07a0005180 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a000c2e0 con 0x7f07a0003970 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 0x7f07a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b259c640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07a0003970 0x7f07a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b2d9d640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a0005180 0x7f07a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07a0003970 msgr2=0x7f07a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07a0003970 0x7f07a0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.823 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a0005180 msgr2=0x7f07a000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a0005180 0x7f07a000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2025519375 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0003520 con 0x7f07a00047c0 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.821+0000 7f07b2d9d640 1 -- 172.21.15.124:0/2025519375 reap_dead start 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.822+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 0x7f07a0004c40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f079c001750 tx=0x7f07a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618471b9767c36bf server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.822+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c0052d0 con 0x7f07a00047c0 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.822+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c02e050 con 0x7f07a00047c0 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.822+0000 7f07b1d9b640 1 -- 172.21.15.124:0/2025519375 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c003830 con 0x7f07a00047c0 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 msgr2=0x7f07a0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 0x7f07a0004c40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f079c001750 tx=0x7f07a00057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2025519375 shutdown_connections 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2025519375 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07a00047c0 0x7f07a0004c40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2025519375 >> 172.21.15.124:0/2025519375 conn(0x7f07b4072020 msgr2=0x7f07b4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2025519375 shutdown_connections 2023-12-13T23:55:53.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.823+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2025519375 wait complete. 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.824+0000 7f07b8ffb640 1 Processor -- start 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.824+0000 7f07b8ffb640 1 -- start start 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a00047c0 0x7f07b41561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b4145160 0x7f07b4145610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4148ba0 con 0x7f07b41493a0 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b4148e80 con 0x7f07a00047c0 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b8ffb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b40cd050 con 0x7f07b4145160 2023-12-13T23:55:53.827 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43426/0 (socket says 172.21.15.124:43426) 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2964993780 learned_addr learned my addr 172.21.15.124:0/2964993780 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b259c640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a00047c0 0x7f07b41561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07b2d9d640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b4145160 0x7f07b4145610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b4145160 msgr2=0x7f07b4145610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b4145160 0x7f07b4145610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a00047c0 msgr2=0x7f07b41561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a00047c0 0x7f07b41561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.825+0000 7f07a9d9b640 1 -- 172.21.15.124:0/2964993780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b40cd330 con 0x7f07b41493a0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07a9d9b640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f079c005040 tx=0x7f079c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c007a30 con 0x7f07b41493a0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079c0069e0 con 0x7f07b41493a0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079c006d70 con 0x7f07b41493a0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b40cd610 con 0x7f07b41493a0 2023-12-13T23:55:53.828 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.826+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b41f6f90 con 0x7f07b41493a0 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.827+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4069400 con 0x7f07b41493a0 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.827+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f079c0035f0 con 0x7f07b41493a0 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.828+0000 7f07ab7fe640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 0x7f079406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.828+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f079c0abb50 con 0x7f07b41493a0 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.828+0000 7f07b259c640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 0x7f079406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:53.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.828+0000 7f07b259c640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 0x7f079406d680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009a00 tx=0x7f07a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:53.830 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:53.832+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f079c076800 con 0x7f07b41493a0 2023-12-13T23:55:54.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.003+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07b41607d0 con 0x7f07b41493a0 2023-12-13T23:55:54.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.003+0000 7f07ab7fe640 1 -- 172.21.15.124:0/2964993780 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f079c04b020 con 0x7f07b41493a0 2023-12-13T23:55:54.002 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:54.003 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 msgr2=0x7f079406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 0x7f079406d680 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f07a4009a00 tx=0x7f07a4008040 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 msgr2=0x7f07b41f6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f079c005040 tx=0x7f079c001120 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 shutdown_connections 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079406b1c0 0x7f079406d680 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b4145160 0x7f07b4145610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b41493a0 0x7f07b41f6a50 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 --2- 172.21.15.124:0/2964993780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07a00047c0 0x7f07b41561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 >> 172.21.15.124:0/2964993780 conn(0x7f07b4072020 msgr2=0x7f07b415e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 shutdown_connections 2023-12-13T23:55:54.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.005+0000 7f07b8ffb640 1 -- 172.21.15.124:0/2964993780 wait complete. 2023-12-13T23:55:54.014 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:54.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.118+0000 7f71cfa90640 1 Processor -- start 2023-12-13T23:55:54.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.118+0000 7f71cfa90640 1 -- start start 2023-12-13T23:55:54.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cfa90640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71c806ea50 0x7f71c806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cfa90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 0x7f71c8161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cfa90640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71c8161d20 con 0x7f71c80b3ff0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cfa90640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c80d1ee0 con 0x7f71c806ea50 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cfa90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c806f370 con 0x7f71c8160fb0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71ce02d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f71c80b3ff0 0x7f71c80b4480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40822/0 (socket says 172.21.15.124:40822) 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71ce02d640 1 -- 172.21.15.124:0/2945386594 learned_addr learned my addr 172.21.15.124:0/2945386594 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cd82c640 1 --2- 172.21.15.124:0/2945386594 >> v2:172.21.15.203:3300/0 conn(0x7f71c806ea50 0x7f71c806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.119+0000 7f71cd02b640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 0x7f71c8161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884094430 0 0) 0x7f71c8161d20 con 0x7f71c80b3ff0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b4003520 con 0x7f71c80b3ff0 2023-12-13T23:55:54.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cd82c640 1 -- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 msgr2=0x7f71c8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cd82c640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 0x7f71c8161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cd82c640 1 -- 172.21.15.124:0/2945386594 >> v1:172.21.15.124:6789/0 conn(0x7f71c80b3ff0 legacy=0x7f71c80b4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 978421918 0 0) 0x7f71b4003520 con 0x7f71c80b3ff0 2023-12-13T23:55:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cd82c640 1 -- 172.21.15.124:0/2945386594 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c815e760 con 0x7f71c806ea50 2023-12-13T23:55:54.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.120+0000 7f71cd02b640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 0x7f71c8161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cd82c640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c806ea50 0x7f71c806ee30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009d90 tx=0x7f71c402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af9acb4078c38376 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c0001670 con 0x7f71c806ea50 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c402c8a0 con 0x7f71c806ea50 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cc82a640 1 -- 172.21.15.124:0/2945386594 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c402cc10 con 0x7f71c806ea50 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 -- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c806ea50 msgr2=0x7f71c806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c806ea50 0x7f71c806ee30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009d90 tx=0x7f71c402bd70 comp rx=0 tx=0).stop 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 -- 172.21.15.124:0/2945386594 shutdown_connections 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c8160fb0 0x7f71c8161440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 --2- 172.21.15.124:0/2945386594 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c806ea50 0x7f71c806ee30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 -- 172.21.15.124:0/2945386594 >> 172.21.15.124:0/2945386594 conn(0x7f71c806a6b0 msgr2=0x7f71c806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 -- 172.21.15.124:0/2945386594 shutdown_connections 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.121+0000 7f71cfa90640 1 -- 172.21.15.124:0/2945386594 wait complete. 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 Processor -- start 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 -- start start 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c80b3ff0 0x7f71c80b1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c8160fb0 0x7f71c813fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8152530 con 0x7f71c806ea50 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8152810 con 0x7f71c8160fb0 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cfa90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c8152af0 con 0x7f71c80b3ff0 2023-12-13T23:55:54.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cd82c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cd82c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43442/0 (socket says 172.21.15.124:43442) 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.123+0000 7f71cd82c640 1 -- 172.21.15.124:0/30540844 learned_addr learned my addr 172.21.15.124:0/30540844 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd02b640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c80b3ff0 0x7f71c80b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 -- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c80b3ff0 msgr2=0x7f71c80b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c80b3ff0 0x7f71c80b1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 -- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c8160fb0 msgr2=0x7f71c813fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c8160fb0 0x7f71c813fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 -- 172.21.15.124:0/30540844 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b8009400 con 0x7f71c806ea50 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cd82c640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009d90 tx=0x7f71c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c402ce70 con 0x7f71c806ea50 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c4009450 con 0x7f71c806ea50 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c4009a70 con 0x7f71c806ea50 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71c403e650 con 0x7f71c806ea50 2023-12-13T23:55:54.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.124+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c81544b0 con 0x7f71c806ea50 2023-12-13T23:55:54.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.125+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c8153010 con 0x7f71c806ea50 2023-12-13T23:55:54.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.126+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f71c403b050 con 0x7f71c806ea50 2023-12-13T23:55:54.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.126+0000 7f71be7fc640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 0x7f71ac06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.126+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f71c40abe10 con 0x7f71c806ea50 2023-12-13T23:55:54.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.126+0000 7f71cd02b640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 0x7f71ac06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.127+0000 7f71cd02b640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 0x7f71ac06d680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f71c8153c10 tx=0x7f71b8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.130+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f71c4076b60 con 0x7f71c806ea50 2023-12-13T23:55:54.293 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.294+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f71c8069400 con 0x7f71c806ea50 2023-12-13T23:55:54.293 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.294+0000 7f71be7fc640 1 -- 172.21.15.124:0/30540844 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f71c4039020 con 0x7f71c806ea50 2023-12-13T23:55:54.293 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 msgr2=0x7f71ac06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 0x7f71ac06d680 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f71c8153c10 tx=0x7f71b8008040 comp rx=0 tx=0).stop 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 msgr2=0x7f71c8151f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f71c4009d90 tx=0x7f71c4004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 shutdown_connections 2023-12-13T23:55:54.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71ac06b1c0 0x7f71ac06d680 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c8160fb0 0x7f71c813fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c80b3ff0 0x7f71c80b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 --2- 172.21.15.124:0/30540844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c806ea50 0x7f71c8151f50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 >> 172.21.15.124:0/30540844 conn(0x7f71c806a6b0 msgr2=0x7f71c80b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 shutdown_connections 2023-12-13T23:55:54.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.296+0000 7f71cfa90640 1 -- 172.21.15.124:0/30540844 wait complete. 2023-12-13T23:55:54.306 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:54.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.410+0000 7fe9316e0640 1 Processor -- start 2023-12-13T23:55:54.409 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.410+0000 7fe9316e0640 1 -- start start 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe9316e0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe92c15d2d0 0x7fe92c0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe9316e0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe92c0ce150 0x7fe92c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe9316e0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c0d1bb0 con 0x7fe92c0d2040 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe9316e0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c15d6b0 con 0x7fe92c0ce150 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe9316e0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0ce700 con 0x7fe92c15d2d0 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe92c0ce150 0x7fe92c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe92c0ce150 0x7fe92c0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43456/0 (socket says 172.21.15.124:43456) 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 -- 172.21.15.124:0/1067290562 learned_addr learned my addr 172.21.15.124:0/1067290562 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92affd640 1 --2- 172.21.15.124:0/1067290562 >> v2:172.21.15.203:3300/0 conn(0x7fe92c15d2d0 0x7fe92c0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 -- 172.21.15.124:0/1067290562 >> v2:172.21.15.203:3300/0 conn(0x7fe92c15d2d0 msgr2=0x7fe92c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 --2- 172.21.15.124:0/1067290562 >> v2:172.21.15.203:3300/0 conn(0x7fe92c15d2d0 0x7fe92c0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 -- 172.21.15.124:0/1067290562 >> v1:172.21.15.203:6789/0 conn(0x7fe92c0d2040 legacy=0x7fe92c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.411+0000 7fe92b7fe640 1 -- 172.21.15.124:0/1067290562 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0d1280 con 0x7fe92c0ce150 2023-12-13T23:55:54.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.412+0000 7fe92b7fe640 1 --2- 172.21.15.124:0/1067290562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0ce150 0x7fe92c0b51a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe92c06c5d0 tx=0x7fe92402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d2464e14c254fb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.412+0000 7fe92a7fc640 1 -- 172.21.15.124:0/1067290562 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92402cb20 con 0x7fe92c0ce150 2023-12-13T23:55:54.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.412+0000 7fe92a7fc640 1 -- 172.21.15.124:0/1067290562 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92402cc80 con 0x7fe92c0ce150 2023-12-13T23:55:54.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.412+0000 7fe92a7fc640 1 -- 172.21.15.124:0/1067290562 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9240343f0 con 0x7fe92c0ce150 2023-12-13T23:55:54.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 -- 172.21.15.124:0/1067290562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0ce150 msgr2=0x7fe92c0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 --2- 172.21.15.124:0/1067290562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0ce150 0x7fe92c0b51a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe92c06c5d0 tx=0x7fe92402bf80 comp rx=0 tx=0).stop 2023-12-13T23:55:54.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 -- 172.21.15.124:0/1067290562 shutdown_connections 2023-12-13T23:55:54.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 --2- 172.21.15.124:0/1067290562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0ce150 0x7fe92c0b51a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 --2- 172.21.15.124:0/1067290562 >> v2:172.21.15.203:3300/0 conn(0x7fe92c15d2d0 0x7fe92c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 -- 172.21.15.124:0/1067290562 >> 172.21.15.124:0/1067290562 conn(0x7fe92c072050 msgr2=0x7fe92c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 -- 172.21.15.124:0/1067290562 shutdown_connections 2023-12-13T23:55:54.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.413+0000 7fe9316e0640 1 -- 172.21.15.124:0/1067290562 wait complete. 2023-12-13T23:55:54.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.414+0000 7fe9316e0640 1 Processor -- start 2023-12-13T23:55:54.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.414+0000 7fe9316e0640 1 -- start start 2023-12-13T23:55:54.416 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe92c0ce150 0x7fe92c0c9130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.416 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe92c15d2d0 0x7fe92c0b0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0aa850 con 0x7fe92c0d2040 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c1f3a40 con 0x7fe92c15d2d0 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c1f3480 con 0x7fe92c0ce150 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43462/0 (socket says 172.21.15.124:43462) 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 -- 172.21.15.124:0/3669860843 learned_addr learned my addr 172.21.15.124:0/3669860843 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe92b7fe640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe92c15d2d0 0x7fe92c0b0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe92affd640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe92c0ce150 0x7fe92c0c9130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 -- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe92c0ce150 msgr2=0x7fe92c0c9130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe92c0ce150 0x7fe92c0c9130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 -- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe92c15d2d0 msgr2=0x7fe92c0b0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe92c15d2d0 0x7fe92c0b0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 -- 172.21.15.124:0/3669860843 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe924009b30 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe923fff640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c450 tx=0x7fe91400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9140052d0 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe914005430 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c1f37c0 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe914012870 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.415+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c0c9670 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.416+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c0c9830 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.417+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe9140129d0 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.417+0000 7fe9237fe640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 0x7fe90c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.417+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe914089390 con 0x7fe92c0d2040 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.417+0000 7fe92affd640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 0x7fe90c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.418+0000 7fe92affd640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 0x7fe90c06d680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009aa0 tx=0x7fe91c009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.421+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe914054180 con 0x7fe92c0d2040 2023-12-13T23:55:54.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.588+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe92c069400 con 0x7fe92c0d2040 2023-12-13T23:55:54.587 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.589+0000 7fe9237fe640 1 -- 172.21.15.124:0/3669860843 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe9140577a0 con 0x7fe92c0d2040 2023-12-13T23:55:54.588 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 msgr2=0x7fe90c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 0x7fe90c06d680 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe91c009aa0 tx=0x7fe91c009190 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 msgr2=0x7fe92c1f2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fe91400c450 tx=0x7fe91400c910 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 shutdown_connections 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe90c06b1c0 0x7fe90c06d680 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe92c15d2d0 0x7fe92c0b0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe92c0d2040 0x7fe92c1f2ac0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 --2- 172.21.15.124:0/3669860843 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe92c0ce150 0x7fe92c0c9130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 >> 172.21.15.124:0/3669860843 conn(0x7fe92c072050 msgr2=0x7fe92c15f800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 shutdown_connections 2023-12-13T23:55:54.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.591+0000 7fe9316e0640 1 -- 172.21.15.124:0/3669860843 wait complete. 2023-12-13T23:55:54.600 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:54.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.702+0000 7fdb2e11d640 1 Processor -- start 2023-12-13T23:55:54.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.702+0000 7fdb2e11d640 1 -- start start 2023-12-13T23:55:54.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb2e11d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb280b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb2e11d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2815e2a0 con 0x7fdb2815abf0 2023-12-13T23:55:54.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb2e11d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb2815b240 con 0x7fdb2815e730 2023-12-13T23:55:54.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb2e11d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2815b520 con 0x7fdb28143490 2023-12-13T23:55:54.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb27fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdb2815abf0 0x7fdb280b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40826/0 (socket says 172.21.15.124:40826) 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb27fff640 1 -- 172.21.15.124:0/2387192433 learned_addr learned my addr 172.21.15.124:0/2387192433 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.703+0000 7fdb277fe640 1 --2- 172.21.15.124:0/2387192433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb280b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197760027 0 0) 0x7fdb2815e2a0 con 0x7fdb2815abf0 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb10003520 con 0x7fdb2815abf0 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319816959 0 0) 0x7fdb2815b240 con 0x7fdb2815e730 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb2815e2a0 con 0x7fdb2815e730 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb277fe640 1 -- 172.21.15.124:0/2387192433 >> v1:172.21.15.203:6789/0 conn(0x7fdb2815e730 legacy=0x7fdb2815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb277fe640 1 -- 172.21.15.124:0/2387192433 >> v1:172.21.15.124:6789/0 conn(0x7fdb2815abf0 legacy=0x7fdb280b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.704+0000 7fdb277fe640 1 -- 172.21.15.124:0/2387192433 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2815dc50 con 0x7fdb28143490 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb277fe640 1 --2- 172.21.15.124:0/2387192433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb280b0cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097d0 tx=0x7fdb1802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bdf9d5cfc2c7f50 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18031890 con 0x7fdb28143490 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb180319f0 con 0x7fdb28143490 2023-12-13T23:55:54.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb267fc640 1 -- 172.21.15.124:0/2387192433 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb18031d60 con 0x7fdb28143490 2023-12-13T23:55:54.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2387192433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 msgr2=0x7fdb280b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2387192433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb280b0cf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdb180097d0 tx=0x7fdb1802b930 comp rx=0 tx=0).stop 2023-12-13T23:55:54.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2387192433 shutdown_connections 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2387192433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb280b0cf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2387192433 >> 172.21.15.124:0/2387192433 conn(0x7fdb2806a4d0 msgr2=0x7fdb2806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2387192433 shutdown_connections 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.705+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2387192433 wait complete. 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.706+0000 7fdb2e11d640 1 Processor -- start 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 -- start start 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb28159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb2815e730 0x7fdb280a9b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280a9710 con 0x7fdb2815abf0 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280a8a90 con 0x7fdb2815e730 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb2e11d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb280a8d70 con 0x7fdb28143490 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb26ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb26ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43466/0 (socket says 172.21.15.124:43466) 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.707+0000 7fdb26ffd640 1 -- 172.21.15.124:0/2077260135 learned_addr learned my addr 172.21.15.124:0/2077260135 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb277fe640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb28159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb27fff640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb2815e730 0x7fdb280a9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 -- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 msgr2=0x7fdb28159530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb28159530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 -- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb2815e730 msgr2=0x7fdb280a9b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb2815e730 0x7fdb280a9b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 -- 172.21.15.124:0/2077260135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb18009400 con 0x7fdb2815abf0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb26ffd640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00c500 tx=0x7fdb1c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c0052d0 con 0x7fdb2815abf0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb1c005450 con 0x7fdb2815abf0 2023-12-13T23:55:54.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb280a9050 con 0x7fdb2815abf0 2023-12-13T23:55:54.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb1c012870 con 0x7fdb2815abf0 2023-12-13T23:55:54.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.708+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb2813cd80 con 0x7fdb2815abf0 2023-12-13T23:55:54.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.709+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb280a9850 con 0x7fdb2815abf0 2023-12-13T23:55:54.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.709+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdb1c0129d0 con 0x7fdb2815abf0 2023-12-13T23:55:54.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.710+0000 7fdb0ffff640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 0x7fdb1406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:54.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.710+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdb1c089540 con 0x7fdb2815abf0 2023-12-13T23:55:54.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.710+0000 7fdb277fe640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 0x7fdb1406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:54.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.711+0000 7fdb277fe640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 0x7fdb1406d680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb18036000 tx=0x7fdb1803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:54.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.714+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdb280a9850 con 0x7fdb2815abf0 2023-12-13T23:55:54.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.892+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdb280a9850 con 0x7fdb2815abf0 2023-12-13T23:55:54.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.893+0000 7fdb0ffff640 1 -- 172.21.15.124:0/2077260135 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdb280a9850 con 0x7fdb2815abf0 2023-12-13T23:55:54.891 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 msgr2=0x7fdb1406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 0x7fdb1406d680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdb18036000 tx=0x7fdb1803c040 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 msgr2=0x7fdb2813c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fdb1c00c500 tx=0x7fdb1c00c9c0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 shutdown_connections 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb1406b1c0 0x7fdb1406d680 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdb2815e730 0x7fdb280a9b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb2815abf0 0x7fdb2813c840 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 --2- 172.21.15.124:0/2077260135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdb28143490 0x7fdb28159530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 >> 172.21.15.124:0/2077260135 conn(0x7fdb2806a4d0 msgr2=0x7fdb2806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:54.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 shutdown_connections 2023-12-13T23:55:54.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:54.895+0000 7fdb2e11d640 1 -- 172.21.15.124:0/2077260135 wait complete. 2023-12-13T23:55:54.904 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:55:54.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:54.904 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2023-12-13T23:55:54.931 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:55:54.931 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:55:54.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:54.932 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-13T23:55:54.986 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:54.986 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:55.005 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:55.005 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:55.064 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:55.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.166+0000 7f07b7010640 1 Processor -- start 2023-12-13T23:55:55.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.166+0000 7f07b7010640 1 -- start start 2023-12-13T23:55:55.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b7010640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07b00ceb70 0x7f07b00cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b7010640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f07b00b4370 0x7f07b00b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.166 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b7010640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b0071770 con 0x7f07b0144980 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b7010640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b015ced0 con 0x7f07b00b4370 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b7010640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b00b4ce0 con 0x7f07b00ceb70 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f07b00b4370 0x7f07b00b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f07b00b4370 0x7f07b00b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43480/0 (socket says 172.21.15.124:43480) 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 -- 172.21.15.124:0/3988962532 learned_addr learned my addr 172.21.15.124:0/3988962532 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07affff640 1 --2- 172.21.15.124:0/3988962532 >> v2:172.21.15.203:3300/0 conn(0x7f07b00ceb70 0x7f07b00cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 -- 172.21.15.124:0/3988962532 >> v2:172.21.15.203:3300/0 conn(0x7f07b00ceb70 msgr2=0x7f07b00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 --2- 172.21.15.124:0/3988962532 >> v2:172.21.15.203:3300/0 conn(0x7f07b00ceb70 0x7f07b00cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 -- 172.21.15.124:0/3988962532 >> v1:172.21.15.203:6789/0 conn(0x7f07b0144980 legacy=0x7f07b015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.167+0000 7f07b55ad640 1 -- 172.21.15.124:0/3988962532 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b00ccdc0 con 0x7f07b00b4370 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.168+0000 7f07b55ad640 1 --2- 172.21.15.124:0/3988962532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b00b4370 0x7f07b00b47a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f07b015b600 tx=0x7f07a802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7016cfb213cc7f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.168+0000 7f07af7fe640 1 -- 172.21.15.124:0/3988962532 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a802ca80 con 0x7f07b00b4370 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.168+0000 7f07af7fe640 1 -- 172.21.15.124:0/3988962532 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a802cbe0 con 0x7f07b00b4370 2023-12-13T23:55:55.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.168+0000 7f07af7fe640 1 -- 172.21.15.124:0/3988962532 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80343f0 con 0x7f07b00b4370 2023-12-13T23:55:55.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 -- 172.21.15.124:0/3988962532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b00b4370 msgr2=0x7f07b00b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 --2- 172.21.15.124:0/3988962532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b00b4370 0x7f07b00b47a0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f07b015b600 tx=0x7f07a802bee0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 -- 172.21.15.124:0/3988962532 shutdown_connections 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 --2- 172.21.15.124:0/3988962532 >> v2:172.21.15.203:3300/0 conn(0x7f07b00ceb70 0x7f07b00cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 --2- 172.21.15.124:0/3988962532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b00b4370 0x7f07b00b47a0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 -- 172.21.15.124:0/3988962532 >> 172.21.15.124:0/3988962532 conn(0x7f07b006a4d0 msgr2=0x7f07b006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 -- 172.21.15.124:0/3988962532 shutdown_connections 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.169+0000 7f07b7010640 1 -- 172.21.15.124:0/3988962532 wait complete. 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.170+0000 7f07b7010640 1 Processor -- start 2023-12-13T23:55:55.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.170+0000 7f07b7010640 1 -- start start 2023-12-13T23:55:55.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 0x7f07b013fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 0x7f07b00c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00c6820 con 0x7f07b0144980 2023-12-13T23:55:55.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0140090 con 0x7f07b00ceb70 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00c5c10 con 0x7f07b00b4370 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43488/0 (socket says 172.21.15.124:43488) 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 -- 172.21.15.124:0/2562708741 learned_addr learned my addr 172.21.15.124:0/2562708741 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07affff640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 0x7f07b00c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b4dac640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 0x7f07b013fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 -- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 msgr2=0x7f07b013fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 0x7f07b013fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 -- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 msgr2=0x7f07b00c7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 0x7f07b00c7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 -- 172.21.15.124:0/2562708741 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a8009b30 con 0x7f07b0144980 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07affff640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 0x7f07b00c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b4dac640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 0x7f07b013fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b55ad640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f07a8036040 tx=0x7f07a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8009510 con 0x7f07b0144980 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a803e450 con 0x7f07b0144980 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b00c5ef0 con 0x7f07b0144980 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a803e7e0 con 0x7f07b0144980 2023-12-13T23:55:55.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.171+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b00c64b0 con 0x7f07b0144980 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.172+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0069400 con 0x7f07b0144980 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.173+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f07a802cd50 con 0x7f07b0144980 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.173+0000 7f07ad7fa640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 0x7f079c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.173+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f07a80ac280 con 0x7f07b0144980 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.174+0000 7f07b4dac640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 0x7f079c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.174+0000 7f07b4dac640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 0x7f079c06d6b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f07a000a830 tx=0x7f07a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.178+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f07a8076f40 con 0x7f07b0144980 2023-12-13T23:55:55.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.348+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07b015b410 con 0x7f07b0144980 2023-12-13T23:55:55.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.349+0000 7f07ad7fa640 1 -- 172.21.15.124:0/2562708741 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16815 (secure 0 0 0) 0x7f07a8033e80 con 0x7f07b0144980 2023-12-13T23:55:55.348 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:55.348 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":23,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:53.877550+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 msgr2=0x7f079c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 0x7f079c06d6b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f07a000a830 tx=0x7f07a0008040 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 msgr2=0x7f07b00c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f07a8036040 tx=0x7f07a8004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 shutdown_connections 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f079c06b1f0 0x7f079c06d6b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07b00ceb70 0x7f07b00c7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07b0144980 0x7f07b00c7d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 --2- 172.21.15.124:0/2562708741 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07b00b4370 0x7f07b013fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 >> 172.21.15.124:0/2562708741 conn(0x7f07b006a4d0 msgr2=0x7f07b0143c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 shutdown_connections 2023-12-13T23:55:55.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.351+0000 7f07b7010640 1 -- 172.21.15.124:0/2562708741 wait complete. 2023-12-13T23:55:55.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:55.361 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:55.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.464+0000 7fe8d43b6640 1 Processor -- start 2023-12-13T23:55:55.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.464+0000 7fe8d43b6640 1 -- start start 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.465+0000 7fe8d43b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8cc0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.465+0000 7fe8d43b6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc0b49f0 con 0x7fe8cc150560 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.465+0000 7fe8d43b6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc0b3560 con 0x7fe8cc150940 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.465+0000 7fe8d43b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0b3de0 con 0x7fe8cc15d910 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d2152640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe8cc150560 0x7fe8cc0b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40840/0 (socket says 172.21.15.124:40840) 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d2152640 1 -- 172.21.15.124:0/412108629 learned_addr learned my addr 172.21.15.124:0/412108629 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:55.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d2953640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8cc0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025990313 0 0) 0x7fe8cc0b49f0 con 0x7fe8cc150560 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b8003520 con 0x7fe8cc150560 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 81268165 0 0) 0x7fe8b8003520 con 0x7fe8cc150560 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 msgr2=0x7fe8cc0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8cc0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 >> v1:172.21.15.203:6789/0 conn(0x7fe8cc150940 legacy=0x7fe8cc15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc0b40c0 con 0x7fe8cc150560 2023-12-13T23:55:55.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fe8c0001670 con 0x7fe8cc150560 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 >> v1:172.21.15.124:6789/0 conn(0x7fe8cc150560 legacy=0x7fe8cc0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8b80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8003f20 0x7fe8b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8b8004d00 0x7fe8b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc0b3de0 con 0x7fe8b8004d00 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800cb70 con 0x7fe8b8003f20 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8b800cd30 con 0x7fe8cc15d910 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d2953640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8b8004d00 0x7fe8b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d2152640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8b80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.466+0000 7fe8d1951640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8003f20 0x7fe8b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 -- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 msgr2=0x7fe8b80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8b80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 -- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8003f20 msgr2=0x7fe8b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8003f20 0x7fe8b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 -- 172.21.15.124:0/412108629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b8003520 con 0x7fe8b8004d00 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2152640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc15d910 0x7fe8b80039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d2953640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8b8004d00 0x7fe8b8005180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009fc0 tx=0x7fe8b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1535991c0ed7ed80 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c402cc20 con 0x7fe8b8004d00 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c400f040 con 0x7fe8b8004d00 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.467+0000 7fe8d1150640 1 -- 172.21.15.124:0/412108629 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4034440 con 0x7fe8b8004d00 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 -- 172.21.15.124:0/412108629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8b8004d00 msgr2=0x7fe8b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/412108629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8b8004d00 0x7fe8b8005180 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4009fc0 tx=0x7fe8b80057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d2953640 1 -- 172.21.15.124:0/412108629 reap_dead start 2023-12-13T23:55:55.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 -- 172.21.15.124:0/412108629 shutdown_connections 2023-12-13T23:55:55.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 -- 172.21.15.124:0/412108629 >> 172.21.15.124:0/412108629 conn(0x7fe8cc06a4d0 msgr2=0x7fe8cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 -- 172.21.15.124:0/412108629 shutdown_connections 2023-12-13T23:55:55.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.468+0000 7fe8d43b6640 1 -- 172.21.15.124:0/412108629 wait complete. 2023-12-13T23:55:55.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.469+0000 7fe8d43b6640 1 Processor -- start 2023-12-13T23:55:55.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.469+0000 7fe8d43b6640 1 -- start start 2023-12-13T23:55:55.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 0x7fe8cc1f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 0x7fe8cc0cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc1f2e50 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc152d90 con 0x7fe8b8004d00 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d43b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8cc1527d0 con 0x7fe8cc1522c0 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43510/0 (socket says 172.21.15.124:43510) 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 -- 172.21.15.124:0/1753256982 learned_addr learned my addr 172.21.15.124:0/1753256982 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d2953640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 0x7fe8cc0cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d2152640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 0x7fe8cc1f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 -- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 msgr2=0x7fe8cc0cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.475 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 0x7fe8cc0cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 -- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 msgr2=0x7fe8cc1f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 0x7fe8cc1f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d1951640 1 -- 172.21.15.124:0/1753256982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc152ab0 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d2953640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 0x7fe8cc0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.470+0000 7fe8d2152640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 0x7fe8cc1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8d1951640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800f820 tx=0x7fe8c800fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c80050d0 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c8005230 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc15ae50 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c800d730 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.471+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc14a840 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.472+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc14a9d0 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.472+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe8c800d890 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.472+0000 7fe8beffd640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 0x7fe8b006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.473+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe8c8089a00 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.473+0000 7fe8d2152640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 0x7fe8b006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.473+0000 7fe8d2152640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 0x7fe8b006d680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00017a0 tx=0x7fe8c0001190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.477+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe8c80546b0 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.643+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8cc069400 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.643+0000 7fe8beffd640 1 -- 172.21.15.124:0/1753256982 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8c8057cd0 con 0x7fe8cc14a3c0 2023-12-13T23:55:55.642 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 msgr2=0x7fe8b006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 0x7fe8b006d680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe8c00017a0 tx=0x7fe8c0001190 comp rx=0 tx=0).stop 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 msgr2=0x7fe8cc151d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe8c800f820 tx=0x7fe8c800fce0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 shutdown_connections 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe8b006b1c0 0x7fe8b006d680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe8cc1522c0 0x7fe8cc0cd8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe8cc14a3c0 0x7fe8cc151d80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 --2- 172.21.15.124:0/1753256982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe8b8004d00 0x7fe8cc1f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 >> 172.21.15.124:0/1753256982 conn(0x7fe8cc06a4d0 msgr2=0x7fe8cc0b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 shutdown_connections 2023-12-13T23:55:55.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.645+0000 7fe8d43b6640 1 -- 172.21.15.124:0/1753256982 wait complete. 2023-12-13T23:55:55.654 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:55.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.757+0000 7fc4cb70f640 1 Processor -- start 2023-12-13T23:55:55.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.757+0000 7fc4cb70f640 1 -- start start 2023-12-13T23:55:55.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4cb70f640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4c4143490 0x7fc4c40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4cb70f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 0x7fc4c40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4cb70f640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4c415e2a0 con 0x7fc4c415e730 2023-12-13T23:55:55.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4cb70f640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c415b520 con 0x7fc4c4143490 2023-12-13T23:55:55.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4cb70f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c4143870 con 0x7fc4c415abf0 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4c94ab640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4c4143490 0x7fc4c40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4c8caa640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc4c415e730 0x7fc4c415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40846/0 (socket says 172.21.15.124:40846) 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4c94ab640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4c4143490 0x7fc4c40b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43516/0 (socket says 172.21.15.124:43516) 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.758+0000 7fc4c8caa640 1 -- 172.21.15.124:0/2187110868 learned_addr learned my addr 172.21.15.124:0/2187110868 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c94ab640 1 -- 172.21.15.124:0/2187110868 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 msgr2=0x7fc4c40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c9cac640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 0x7fc4c40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c94ab640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 0x7fc4c40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c94ab640 1 -- 172.21.15.124:0/2187110868 >> v1:172.21.15.124:6789/0 conn(0x7fc4c415e730 legacy=0x7fc4c415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4bbfff640 1 -- 172.21.15.124:0/2187110868 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 417919879 0 0) 0x7fc4c415e2a0 con 0x7fc4c415e730 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c94ab640 1 -- 172.21.15.124:0/2187110868 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c415b240 con 0x7fc4c4143490 2023-12-13T23:55:55.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c9cac640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 0x7fc4c40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4c94ab640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c4143490 0x7fc4c40b0cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009d90 tx=0x7fc4c002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05afb3888dd49eb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.759+0000 7fc4bbfff640 1 -- 172.21.15.124:0/2187110868 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c002cc50 con 0x7fc4c4143490 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4bbfff640 1 -- 172.21.15.124:0/2187110868 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c002cdb0 con 0x7fc4c4143490 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4bbfff640 1 -- 172.21.15.124:0/2187110868 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0034660 con 0x7fc4c4143490 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2187110868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c4143490 msgr2=0x7fc4c40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c4143490 0x7fc4c40b0cf0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0009d90 tx=0x7fc4c002bea0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2187110868 shutdown_connections 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c415abf0 0x7fc4c40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2187110868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c4143490 0x7fc4c40b0cf0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2187110868 >> 172.21.15.124:0/2187110868 conn(0x7fc4c406a4d0 msgr2=0x7fc4c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2187110868 shutdown_connections 2023-12-13T23:55:55.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.760+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2187110868 wait complete. 2023-12-13T23:55:55.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.761+0000 7fc4cb70f640 1 Processor -- start 2023-12-13T23:55:55.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.761+0000 7fc4cb70f640 1 -- start start 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c4143490 0x7fc4c40ade00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4c415e730 0x7fc4c40ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c413ede0 con 0x7fc4c415abf0 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40b0930 con 0x7fc4c415e730 2023-12-13T23:55:55.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4cb70f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c40b0c10 con 0x7fc4c4143490 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43528/0 (socket says 172.21.15.124:43528) 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 -- 172.21.15.124:0/2291328418 learned_addr learned my addr 172.21.15.124:0/2291328418 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 -- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c4143490 msgr2=0x7fc4c40ade00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c9cac640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4c415e730 0x7fc4c40ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c94ab640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c4143490 0x7fc4c40ade00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c4143490 0x7fc4c40ade00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 -- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4c415e730 msgr2=0x7fc4c40ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4c415e730 0x7fc4c40ac210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.762+0000 7fc4c8caa640 1 -- 172.21.15.124:0/2291328418 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c0009a70 con 0x7fc4c415abf0 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4c8caa640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c430 tx=0x7fc4b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40074e0 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b4007660 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c40b0ef0 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b4004a20 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.763+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c40b14b0 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.764+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c4069400 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.764+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc4b4004b80 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.765+0000 7fc4b9ffb640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 0x7fc4a806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.765+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc4b4089910 con 0x7fc4c415abf0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.765+0000 7fc4c94ab640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 0x7fc4a806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:55.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.765+0000 7fc4c94ab640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 0x7fc4a806d680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:55.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.769+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc4b40545c0 con 0x7fc4c415abf0 2023-12-13T23:55:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.935+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc4c40b2820 con 0x7fc4c415abf0 2023-12-13T23:55:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.936+0000 7fc4b9ffb640 1 -- 172.21.15.124:0/2291328418 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc4b4057be0 con 0x7fc4c415abf0 2023-12-13T23:55:55.935 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 msgr2=0x7fc4a806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 0x7fc4a806d680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0036000 tx=0x7fc4c0030000 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 msgr2=0x7fc4c40abcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc4b400c430 tx=0x7fc4b400c8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 shutdown_connections 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc4a806b1c0 0x7fc4a806d680 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc4c415e730 0x7fc4c40ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4c415abf0 0x7fc4c40abcd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 --2- 172.21.15.124:0/2291328418 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4c4143490 0x7fc4c40ade00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 >> 172.21.15.124:0/2291328418 conn(0x7fc4c406a4d0 msgr2=0x7fc4c415c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 shutdown_connections 2023-12-13T23:55:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:55.938+0000 7fc4cb70f640 1 -- 172.21.15.124:0/2291328418 wait complete. 2023-12-13T23:55:55.947 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:56.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.052+0000 7f2149189640 1 Processor -- start 2023-12-13T23:55:56.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.052+0000 7f2149189640 1 -- start start 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f2149189640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f214406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f2149189640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21440ad640 0x7f214406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f2149189640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21440adae0 con 0x7f21440aeb80 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f2149189640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21440addc0 con 0x7f21440ad640 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f2149189640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ae0a0 con 0x7f214415d1d0 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21440ad640 0x7f214406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21440ad640 0x7f214406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43534/0 (socket says 172.21.15.124:43534) 2023-12-13T23:55:56.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 -- 172.21.15.124:0/2964604127 learned_addr learned my addr 172.21.15.124:0/2964604127 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 -- 172.21.15.124:0/2964604127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 msgr2=0x7f214406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 --2- 172.21.15.124:0/2964604127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f214406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 -- 172.21.15.124:0/2964604127 >> v1:172.21.15.203:6789/0 conn(0x7f21440aeb80 legacy=0x7f2144160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.053+0000 7f214259c640 1 -- 172.21.15.124:0/2964604127 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21440ae380 con 0x7f21440ad640 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.054+0000 7f214259c640 1 --2- 172.21.15.124:0/2964604127 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440ad640 0x7f214406ba20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2138009870 tx=0x7f213802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64d35dff4e345860 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.054+0000 7f2141d9b640 1 -- 172.21.15.124:0/2964604127 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138031640 con 0x7f21440ad640 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.054+0000 7f2141d9b640 1 -- 172.21.15.124:0/2964604127 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21380317a0 con 0x7f21440ad640 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.054+0000 7f2141d9b640 1 -- 172.21.15.124:0/2964604127 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138031b30 con 0x7f21440ad640 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 -- 172.21.15.124:0/2964604127 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440ad640 msgr2=0x7f214406ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 --2- 172.21.15.124:0/2964604127 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440ad640 0x7f214406ba20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2138009870 tx=0x7f213802b8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 -- 172.21.15.124:0/2964604127 shutdown_connections 2023-12-13T23:55:56.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 --2- 172.21.15.124:0/2964604127 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440ad640 0x7f214406ba20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 --2- 172.21.15.124:0/2964604127 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f214406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 -- 172.21.15.124:0/2964604127 >> 172.21.15.124:0/2964604127 conn(0x7f2144070e70 msgr2=0x7f2144071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 -- 172.21.15.124:0/2964604127 shutdown_connections 2023-12-13T23:55:56.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.055+0000 7f2149189640 1 -- 172.21.15.124:0/2964604127 wait complete. 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.056+0000 7f2149189640 1 Processor -- start 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.056+0000 7f2149189640 1 -- start start 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 0x7f2144160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f21441528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440cd700 con 0x7f21440aeb80 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21440ccdd0 con 0x7f21440ad640 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2149189640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21440cd0b0 con 0x7f214415d1d0 2023-12-13T23:55:56.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f214259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f214259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43550/0 (socket says 172.21.15.124:43550) 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f214259c640 1 -- 172.21.15.124:0/4260791588 learned_addr learned my addr 172.21.15.124:0/4260791588 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f214359e640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f21441528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.057+0000 7f2142d9d640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 0x7f2144160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 -- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 msgr2=0x7f21441528a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f21441528a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 -- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 msgr2=0x7f2144160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 0x7f2144160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 -- 172.21.15.124:0/4260791588 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21380094d0 con 0x7f21440aeb80 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214359e640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f21441528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f2142d9d640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 0x7f2144160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f214259c640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2138009870 tx=0x7f21380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213803ac60 con 0x7f21440aeb80 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213803adc0 con 0x7f21440aeb80 2023-12-13T23:55:56.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21440cd390 con 0x7f21440aeb80 2023-12-13T23:55:56.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138044630 con 0x7f21440aeb80 2023-12-13T23:55:56.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.058+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214415e590 con 0x7f21440aeb80 2023-12-13T23:55:56.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.059+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2144069400 con 0x7f21440aeb80 2023-12-13T23:55:56.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.059+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f213802f050 con 0x7f21440aeb80 2023-12-13T23:55:56.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.060+0000 7f212b7fe640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 0x7f213006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.060+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f21380aba20 con 0x7f21440aeb80 2023-12-13T23:55:56.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.060+0000 7f2142d9d640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 0x7f213006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.060+0000 7f2142d9d640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 0x7f213006d680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2134007eb0 tx=0x7f21340073c0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.064+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2138076760 con 0x7f21440aeb80 2023-12-13T23:55:56.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.231+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f214415b9b0 con 0x7f21440aeb80 2023-12-13T23:55:56.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.232+0000 7f212b7fe640 1 -- 172.21.15.124:0/4260791588 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2138031e60 con 0x7f21440aeb80 2023-12-13T23:55:56.230 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:56.232 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 msgr2=0x7f213006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.232 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 0x7f213006d680 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2134007eb0 tx=0x7f21340073c0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.232 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 msgr2=0x7f2144152360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f2138009870 tx=0x7f21380046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 shutdown_connections 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f213006b1c0 0x7f213006d680 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214415d1d0 0x7f21441528a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21440aeb80 0x7f2144152360 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.233+0000 7f2149189640 1 --2- 172.21.15.124:0/4260791588 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21440ad640 0x7f2144160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.234+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 >> 172.21.15.124:0/4260791588 conn(0x7f2144070e70 msgr2=0x7f214415bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.234+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 shutdown_connections 2023-12-13T23:55:56.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.234+0000 7f2149189640 1 -- 172.21.15.124:0/4260791588 wait complete. 2023-12-13T23:55:56.243 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:56.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.345+0000 7f112d617640 1 Processor -- start 2023-12-13T23:55:56.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.345+0000 7f112d617640 1 -- start start 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f112d617640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f112815d330 0x7f11280b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f112d617640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11280d2910 0x7f11280d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f112d617640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11280b2b10 con 0x7f11280d2d90 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f112d617640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11280d1590 con 0x7f11280d2910 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f112d617640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11280d1870 con 0x7f112815d330 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f11267fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11280d2910 0x7f11280d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f11267fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11280d2910 0x7f11280d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43564/0 (socket says 172.21.15.124:43564) 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f11267fc640 1 -- 172.21.15.124:0/1981780186 learned_addr learned my addr 172.21.15.124:0/1981780186 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.346+0000 7f1126ffd640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f112815d330 0x7f11280b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f11267fc640 1 -- 172.21.15.124:0/1981780186 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f112815d330 msgr2=0x7f11280b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f11267fc640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f112815d330 0x7f11280b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f11267fc640 1 -- 172.21.15.124:0/1981780186 >> v1:172.21.15.203:6789/0 conn(0x7f11280d2d90 legacy=0x7f11280b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f11267fc640 1 -- 172.21.15.124:0/1981780186 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11280d1b50 con 0x7f11280d2910 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f11267fc640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11280d2910 0x7f11280d31f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f111c009870 tx=0x7f111c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fd2ea0560cf8a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f1125ffb640 1 -- 172.21.15.124:0/1981780186 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c0317c0 con 0x7f11280d2910 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f1125ffb640 1 -- 172.21.15.124:0/1981780186 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c031920 con 0x7f11280d2910 2023-12-13T23:55:56.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.347+0000 7f1125ffb640 1 -- 172.21.15.124:0/1981780186 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c031cb0 con 0x7f11280d2910 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 -- 172.21.15.124:0/1981780186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11280d2910 msgr2=0x7f11280d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11280d2910 0x7f11280d31f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f111c009870 tx=0x7f111c02b860 comp rx=0 tx=0).stop 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 -- 172.21.15.124:0/1981780186 shutdown_connections 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11280d2910 0x7f11280d31f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 --2- 172.21.15.124:0/1981780186 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f112815d330 0x7f11280b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 -- 172.21.15.124:0/1981780186 >> 172.21.15.124:0/1981780186 conn(0x7f1128072020 msgr2=0x7f1128072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 -- 172.21.15.124:0/1981780186 shutdown_connections 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.348+0000 7f112d617640 1 -- 172.21.15.124:0/1981780186 wait complete. 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.349+0000 7f112d617640 1 Processor -- start 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.349+0000 7f112d617640 1 -- start start 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11280d2910 0x7f11281f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11280d2d90 0x7f11281602d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11280cd100 con 0x7f112815d330 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11281f72a0 con 0x7f11280d2910 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f112d617640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11281f3c40 con 0x7f11280d2d90 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43574/0 (socket says 172.21.15.124:43574) 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 -- 172.21.15.124:0/872328593 learned_addr learned my addr 172.21.15.124:0/872328593 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f1126ffd640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11280d2910 0x7f11281f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11267fc640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11280d2d90 0x7f11281602d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 -- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11280d2d90 msgr2=0x7f11281602d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11280d2d90 0x7f11281602d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 -- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11280d2910 msgr2=0x7f11281f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.350+0000 7f11277fe640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11280d2910 0x7f11281f6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f11277fe640 1 -- 172.21.15.124:0/872328593 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f111c0094d0 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f11277fe640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f112000c730 tx=0x7f112000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000aa50 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112000abb0 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112000ce40 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11280cc4f0 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.351+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11280ccab0 con 0x7f112815d330 2023-12-13T23:55:56.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.352+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1128069400 con 0x7f112815d330 2023-12-13T23:55:56.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.352+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1120012420 con 0x7f112815d330 2023-12-13T23:55:56.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.352+0000 7f110f7fe640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 0x7f111406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.353+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1120089f50 con 0x7f112815d330 2023-12-13T23:55:56.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.353+0000 7f1126ffd640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 0x7f111406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.353+0000 7f1126ffd640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 0x7f111406d680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1118006fd0 tx=0x7f1118008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.357+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1120054c00 con 0x7f112815d330 2023-12-13T23:55:56.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.537+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f112815ce70 con 0x7f112815d330 2023-12-13T23:55:56.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.537+0000 7f110f7fe640 1 -- 172.21.15.124:0/872328593 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1120016020 con 0x7f112815d330 2023-12-13T23:55:56.536 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 msgr2=0x7f111406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 0x7f111406d680 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1118006fd0 tx=0x7f1118008040 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 msgr2=0x7f11281f2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f112000c730 tx=0x7f112000cbf0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 shutdown_connections 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f111406b1c0 0x7f111406d680 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112815d330 0x7f11281f2d50 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11280d2d90 0x7f11281602d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 --2- 172.21.15.124:0/872328593 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11280d2910 0x7f11281f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 >> 172.21.15.124:0/872328593 conn(0x7f1128072020 msgr2=0x7f11280cf4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 shutdown_connections 2023-12-13T23:55:56.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.539+0000 7f112d617640 1 -- 172.21.15.124:0/872328593 wait complete. 2023-12-13T23:55:56.549 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:55:56.549 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:55:56.569 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:56.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.670+0000 7fdc51ba9640 1 Processor -- start 2023-12-13T23:55:56.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.670+0000 7fdc51ba9640 1 -- start start 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc51ba9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc51ba9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c15e2a0 con 0x7fdc4c143490 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc51ba9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc4c15b240 con 0x7fdc4c15e730 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc51ba9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc4c15b520 con 0x7fdc4c0b1400 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43576/0 (socket says 172.21.15.124:43576) 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc4b7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdc4c143490 0x7fdc4c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40850/0 (socket says 172.21.15.124:40850) 2023-12-13T23:55:56.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 -- 172.21.15.124:0/4235163106 learned_addr learned my addr 172.21.15.124:0/4235163106 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 -- 172.21.15.124:0/4235163106 >> v1:172.21.15.203:6789/0 conn(0x7fdc4c15e730 legacy=0x7fdc4c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 -- 172.21.15.124:0/4235163106 >> v1:172.21.15.124:6789/0 conn(0x7fdc4c143490 legacy=0x7fdc4c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.671+0000 7fdc42ffd640 1 -- 172.21.15.124:0/4235163106 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc4c15dc50 con 0x7fdc4c0b1400 2023-12-13T23:55:56.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.672+0000 7fdc42ffd640 1 --2- 172.21.15.124:0/4235163106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400bb10 tx=0x7fdc3402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd66dc53227dde3b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.672+0000 7fdc4affd640 1 -- 172.21.15.124:0/4235163106 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34036440 con 0x7fdc4c0b1400 2023-12-13T23:55:56.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.672+0000 7fdc4affd640 1 -- 172.21.15.124:0/4235163106 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc340365a0 con 0x7fdc4c0b1400 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.672+0000 7fdc4affd640 1 -- 172.21.15.124:0/4235163106 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34036930 con 0x7fdc4c0b1400 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 -- 172.21.15.124:0/4235163106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c0b1400 msgr2=0x7fdc4c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/4235163106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdc3400bb10 tx=0x7fdc3402e170 comp rx=0 tx=0).stop 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 -- 172.21.15.124:0/4235163106 shutdown_connections 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/4235163106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c0b1400 0x7fdc4c15a4e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 -- 172.21.15.124:0/4235163106 >> 172.21.15.124:0/4235163106 conn(0x7fdc4c06a4d0 msgr2=0x7fdc4c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 -- 172.21.15.124:0/4235163106 shutdown_connections 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.673+0000 7fdc51ba9640 1 -- 172.21.15.124:0/4235163106 wait complete. 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.674+0000 7fdc51ba9640 1 Processor -- start 2023-12-13T23:55:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.674+0000 7fdc51ba9640 1 -- start start 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdc4c0b1400 0x7fdc4c159a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc4c143490 0x7fdc4c15d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c135390 con 0x7fdc4c15e730 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c071340 con 0x7fdc4c143490 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc51ba9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc4c15b790 con 0x7fdc4c0b1400 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43578/0 (socket says 172.21.15.124:43578) 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 -- 172.21.15.124:0/2508180387 learned_addr learned my addr 172.21.15.124:0/2508180387 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc42ffd640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc4c143490 0x7fdc4c15d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4b7fe640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdc4c0b1400 0x7fdc4c159a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 -- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdc4c0b1400 msgr2=0x7fdc4c159a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdc4c0b1400 0x7fdc4c159a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 -- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc4c143490 msgr2=0x7fdc4c15d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc4c143490 0x7fdc4c15d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 -- 172.21.15.124:0/2508180387 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3400b770 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc4bfff640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400b9c0 tx=0x7fdc4400be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.675+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc4400cba0 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.676+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc4400cd00 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.676+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc4c15ba70 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.676+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc44014680 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.676+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc4c0ad2f0 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.676+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc4c069400 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.677+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdc4401d3b0 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.677+0000 7fdc48ff9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 0x7fdc3006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.677+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdc440899b0 con 0x7fdc4c15e730 2023-12-13T23:55:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.677+0000 7fdc4b7fe640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 0x7fdc3006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.678+0000 7fdc4b7fe640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 0x7fdc3006d680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000f60 tx=0x7fdc3c007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.681+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdc440546f0 con 0x7fdc4c15e730 2023-12-13T23:55:56.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.848+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc4c0b2820 con 0x7fdc4c15e730 2023-12-13T23:55:56.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.849+0000 7fdc48ff9640 1 -- 172.21.15.124:0/2508180387 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16815 (secure 0 0 0) 0x7fdc44019020 con 0x7fdc4c15e730 2023-12-13T23:55:56.848 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:56.848 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":23,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:53.877550+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 msgr2=0x7fdc3006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 0x7fdc3006d680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdc3c000f60 tx=0x7fdc3c007ca0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 msgr2=0x7fdc4c134e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400b9c0 tx=0x7fdc4400be80 comp rx=0 tx=0).stop 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 shutdown_connections 2023-12-13T23:55:56.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdc3006b1c0 0x7fdc3006d680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdc4c143490 0x7fdc4c15d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdc4c15e730 0x7fdc4c134e50 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 --2- 172.21.15.124:0/2508180387 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdc4c0b1400 0x7fdc4c159a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 >> 172.21.15.124:0/2508180387 conn(0x7fdc4c06a4d0 msgr2=0x7fdc4c071af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 shutdown_connections 2023-12-13T23:55:56.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.851+0000 7fdc51ba9640 1 -- 172.21.15.124:0/2508180387 wait complete. 2023-12-13T23:55:56.861 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:56.861 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:56.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.966+0000 7f6c40b61640 1 Processor -- start 2023-12-13T23:55:56.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.966+0000 7f6c40b61640 1 -- start start 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c40b61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6c3c143490 0x7f6c3c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c40b61640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6c3c15e730 0x7f6c3c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c40b61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c3c15e2a0 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c40b61640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c15b520 con 0x7f6c3c15e730 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c40b61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c3c143870 con 0x7f6c3c143490 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6c3c15e730 0x7f6c3c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6c3c15e730 0x7f6c3c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43580/0 (socket says 172.21.15.124:43580) 2023-12-13T23:55:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 -- 172.21.15.124:0/1721480305 learned_addr learned my addr 172.21.15.124:0/1721480305 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c3a59c640 1 --2- 172.21.15.124:0/1721480305 >> v2:172.21.15.203:3300/0 conn(0x7f6c3c143490 0x7f6c3c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 -- 172.21.15.124:0/1721480305 >> v2:172.21.15.203:3300/0 conn(0x7f6c3c143490 msgr2=0x7f6c3c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 --2- 172.21.15.124:0/1721480305 >> v2:172.21.15.203:3300/0 conn(0x7f6c3c143490 0x7f6c3c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 -- 172.21.15.124:0/1721480305 >> v1:172.21.15.203:6789/0 conn(0x7f6c3c15a4e0 legacy=0x7f6c3c15a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.967+0000 7f6c33fff640 1 -- 172.21.15.124:0/1721480305 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c15b240 con 0x7f6c3c15e730 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c33fff640 1 --2- 172.21.15.124:0/1721480305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15e730 0x7f6c3c0b1230 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06e620 tx=0x7f6c2402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34940f70f6e959d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c39d9b640 1 -- 172.21.15.124:0/1721480305 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c2402ee90 con 0x7f6c3c15e730 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c39d9b640 1 -- 172.21.15.124:0/1721480305 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c24036440 con 0x7f6c3c15e730 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c39d9b640 1 -- 172.21.15.124:0/1721480305 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c240367d0 con 0x7f6c3c15e730 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 -- 172.21.15.124:0/1721480305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15e730 msgr2=0x7f6c3c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 --2- 172.21.15.124:0/1721480305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15e730 0x7f6c3c0b1230 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c06e620 tx=0x7f6c2402e0e0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 -- 172.21.15.124:0/1721480305 shutdown_connections 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 --2- 172.21.15.124:0/1721480305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15e730 0x7f6c3c0b1230 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 --2- 172.21.15.124:0/1721480305 >> v2:172.21.15.203:3300/0 conn(0x7f6c3c143490 0x7f6c3c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.968+0000 7f6c40b61640 1 -- 172.21.15.124:0/1721480305 >> 172.21.15.124:0/1721480305 conn(0x7f6c3c06a4d0 msgr2=0x7f6c3c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:56.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.969+0000 7f6c40b61640 1 -- 172.21.15.124:0/1721480305 shutdown_connections 2023-12-13T23:55:56.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.969+0000 7f6c40b61640 1 -- 172.21.15.124:0/1721480305 wait complete. 2023-12-13T23:55:56.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.970+0000 7f6c40b61640 1 Processor -- start 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.970+0000 7f6c40b61640 1 -- start start 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.970+0000 7f6c40b61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3c143490 0x7f6c3c140e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.970+0000 7f6c40b61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6c3c15e730 0x7f6c3c15cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0ac910 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0acbf0 con 0x7f6c3c143490 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3c0aced0 con 0x7f6c3c15e730 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43594/0 (socket says 172.21.15.124:43594) 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 -- 172.21.15.124:0/2100877104 learned_addr learned my addr 172.21.15.124:0/2100877104 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c3a59c640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3c143490 0x7f6c3c140e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c3ad9d640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6c3c15e730 0x7f6c3c15cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 -- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6c3c15e730 msgr2=0x7f6c3c15cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6c3c15e730 0x7f6c3c15cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 -- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3c143490 msgr2=0x7f6c3c140e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3c143490 0x7f6c3c140e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 -- 172.21.15.124:0/2100877104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2400b770 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c33fff640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6c2402e120 tx=0x7f6c240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24034b80 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c24034d00 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c24041680 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c3c0ad1b0 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.971+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c3c0ac5c0 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.972+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3c0ad3f0 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.973+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6c2403d040 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.973+0000 7f6c32ffd640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 0x7f6c1c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:56.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.973+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6c240ae940 con 0x7f6c3c15a4e0 2023-12-13T23:55:56.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.974+0000 7f6c3a59c640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 0x7f6c1c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:56.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.974+0000 7f6c3a59c640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 0x7f6c1c06d6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00aac0 tx=0x7f6c2c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:56.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:56.978+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6c3c0ad3f0 con 0x7f6c3c15a4e0 2023-12-13T23:55:57.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.144+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c3c0ad3f0 con 0x7f6c3c15a4e0 2023-12-13T23:55:57.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.144+0000 7f6c32ffd640 1 -- 172.21.15.124:0/2100877104 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c3c0ad3f0 con 0x7f6c3c15a4e0 2023-12-13T23:55:57.143 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 msgr2=0x7f6c1c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 0x7f6c1c06d6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f6c2c00aac0 tx=0x7f6c2c008040 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 msgr2=0x7f6c3c1e5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6c2402e120 tx=0x7f6c240046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 shutdown_connections 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6c1c06b1f0 0x7f6c1c06d6b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6c3c15e730 0x7f6c3c15cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c3c15a4e0 0x7f6c3c1e5240 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 --2- 172.21.15.124:0/2100877104 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6c3c143490 0x7f6c3c140e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 >> 172.21.15.124:0/2100877104 conn(0x7f6c3c06a4d0 msgr2=0x7f6c3c0ce050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 shutdown_connections 2023-12-13T23:55:57.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.146+0000 7f6c40b61640 1 -- 172.21.15.124:0/2100877104 wait complete. 2023-12-13T23:55:57.156 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:57.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.259+0000 7f768b5f1640 1 Processor -- start 2023-12-13T23:55:57.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768b5f1640 1 -- start start 2023-12-13T23:55:57.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768b5f1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f768406d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768b5f1640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840adae0 con 0x7f768415d1d0 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768b5f1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76840addc0 con 0x7f76840ad640 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768b5f1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840ae0a0 con 0x7f768406bbf0 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768938d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f768415d1d0 0x7f768406b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40854/0 (socket says 172.21.15.124:40854) 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.260+0000 7f768938d640 1 -- 172.21.15.124:0/1756582594 learned_addr learned my addr 172.21.15.124:0/1756582594 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f7689b8e640 1 --2- 172.21.15.124:0/1756582594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f768406d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085278834 0 0) 0x7f76840adae0 con 0x7f768415d1d0 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7670003520 con 0x7f768415d1d0 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734722480 0 0) 0x7f76840addc0 con 0x7f76840ad640 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f7689b8e640 1 -- 172.21.15.124:0/1756582594 >> v1:172.21.15.203:6789/0 conn(0x7f76840ad640 legacy=0x7f768406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f7689b8e640 1 -- 172.21.15.124:0/1756582594 >> v1:172.21.15.124:6789/0 conn(0x7f768415d1d0 legacy=0x7f768406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.261+0000 7f7689b8e640 1 -- 172.21.15.124:0/1756582594 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840ae380 con 0x7f768406bbf0 2023-12-13T23:55:57.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f7689b8e640 1 --2- 172.21.15.124:0/1756582594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f768406d7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f767c009dc0 tx=0x7f767c02bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=225da0bdd85215d3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680001670 con 0x7f768406bbf0 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c02c840 con 0x7f768406bbf0 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f767bfff640 1 -- 172.21.15.124:0/1756582594 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c02cbb0 con 0x7f768406bbf0 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f768b5f1640 1 -- 172.21.15.124:0/1756582594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 msgr2=0x7f768406d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f768b5f1640 1 --2- 172.21.15.124:0/1756582594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f768406d7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f767c009dc0 tx=0x7f767c02bd10 comp rx=0 tx=0).stop 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f768b5f1640 1 -- 172.21.15.124:0/1756582594 shutdown_connections 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f768b5f1640 1 --2- 172.21.15.124:0/1756582594 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f768406d7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.262+0000 7f768b5f1640 1 -- 172.21.15.124:0/1756582594 >> 172.21.15.124:0/1756582594 conn(0x7f7684070e70 msgr2=0x7f7684071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.263+0000 7f768b5f1640 1 -- 172.21.15.124:0/1756582594 shutdown_connections 2023-12-13T23:55:57.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.263+0000 7f768b5f1640 1 -- 172.21.15.124:0/1756582594 wait complete. 2023-12-13T23:55:57.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 Processor -- start 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 -- start start 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f76841f6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 0x7f76841f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76841f7470 con 0x7f768415d1d0 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7684152810 con 0x7f76840ad640 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f768b5f1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76841f34b0 con 0x7f768406bbf0 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f7689b8e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f7689b8e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43606/0 (socket says 172.21.15.124:43606) 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.264+0000 7f7689b8e640 1 -- 172.21.15.124:0/2889447888 learned_addr learned my addr 172.21.15.124:0/2889447888 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f768938d640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f76841f6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7688b8c640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 0x7f76841f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 -- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 msgr2=0x7f76841f6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f76841f6f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 -- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 msgr2=0x7f76841f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 0x7f76841f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 -- 172.21.15.124:0/2889447888 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f767c009a70 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f768938d640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f76841f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7688b8c640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 0x7f76841f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7689b8e640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f767c02c320 tx=0x7f767c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c009450 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c02ce30 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76841f3790 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c03e650 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.265+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76841f3d50 con 0x7f768415d1d0 2023-12-13T23:55:57.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.266+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7684069400 con 0x7f768415d1d0 2023-12-13T23:55:57.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.267+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f767c03b050 con 0x7f768415d1d0 2023-12-13T23:55:57.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.267+0000 7f7679ffb640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 0x7f766806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.267+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f767c0ac160 con 0x7f768415d1d0 2023-12-13T23:55:57.268 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.267+0000 7f768938d640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 0x7f766806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.268 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.268+0000 7f768938d640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 0x7f766806d680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7680000c00 tx=0x7f7680009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.272+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f767c076d90 con 0x7f768415d1d0 2023-12-13T23:55:57.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.437+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76840cc250 con 0x7f768415d1d0 2023-12-13T23:55:57.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.437+0000 7f7679ffb640 1 -- 172.21.15.124:0/2889447888 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f767c07a3b0 con 0x7f768415d1d0 2023-12-13T23:55:57.436 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 msgr2=0x7f766806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 0x7f766806d680 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7680000c00 tx=0x7f7680009040 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 msgr2=0x7f7684158290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f767c02c320 tx=0x7f767c004a40 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 shutdown_connections 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f766806b1c0 0x7f766806d680 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f76840ad640 0x7f76841f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f768415d1d0 0x7f7684158290 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 --2- 172.21.15.124:0/2889447888 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f768406bbf0 0x7f76841f6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 >> 172.21.15.124:0/2889447888 conn(0x7f7684070e70 msgr2=0x7f76840b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 shutdown_connections 2023-12-13T23:55:57.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.439+0000 7f768b5f1640 1 -- 172.21.15.124:0/2889447888 wait complete. 2023-12-13T23:55:57.448 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:57.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.552+0000 7f27e2978640 1 Processor -- start 2023-12-13T23:55:57.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.552+0000 7f27e2978640 1 -- start start 2023-12-13T23:55:57.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27e2978640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f27dc15d330 0x7f27dc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27e2978640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc0b2b10 con 0x7f27dc0d2910 2023-12-13T23:55:57.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27e2978640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27dc0d1590 con 0x7f27dc0d3900 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27e2978640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27dc0d1870 con 0x7f27dc15d330 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27dbfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f27dc15d330 0x7f27dc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27dbfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f27dc15d330 0x7f27dc0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43616/0 (socket says 172.21.15.124:43616) 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27db7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f27dc0d2910 0x7f27dc0d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40864/0 (socket says 172.21.15.124:40864) 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.553+0000 7f27dbfff640 1 -- 172.21.15.124:0/1899014328 learned_addr learned my addr 172.21.15.124:0/1899014328 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27dbfff640 1 -- 172.21.15.124:0/1899014328 >> v1:172.21.15.203:6789/0 conn(0x7f27dc0d3900 legacy=0x7f27dc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27dbfff640 1 -- 172.21.15.124:0/1899014328 >> v1:172.21.15.124:6789/0 conn(0x7f27dc0d2910 legacy=0x7f27dc0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27daffd640 1 -- 172.21.15.124:0/1899014328 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919026424 0 0) 0x7f27dc0b2b10 con 0x7f27dc0d2910 2023-12-13T23:55:57.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27dbfff640 1 -- 172.21.15.124:0/1899014328 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27dc0d1b50 con 0x7f27dc15d330 2023-12-13T23:55:57.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27dbfff640 1 --2- 172.21.15.124:0/1899014328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc15d330 0x7f27dc0b8b80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009870 tx=0x7f27cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1bc63f007be77f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27daffd640 1 -- 172.21.15.124:0/1899014328 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc031870 con 0x7f27dc15d330 2023-12-13T23:55:57.554 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27daffd640 1 -- 172.21.15.124:0/1899014328 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc0319d0 con 0x7f27dc15d330 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.554+0000 7f27daffd640 1 -- 172.21.15.124:0/1899014328 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc031d60 con 0x7f27dc15d330 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 -- 172.21.15.124:0/1899014328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc15d330 msgr2=0x7f27dc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 --2- 172.21.15.124:0/1899014328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc15d330 0x7f27dc0b8b80 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009870 tx=0x7f27cc02b910 comp rx=0 tx=0).stop 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 -- 172.21.15.124:0/1899014328 shutdown_connections 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 --2- 172.21.15.124:0/1899014328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc15d330 0x7f27dc0b8b80 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 -- 172.21.15.124:0/1899014328 >> 172.21.15.124:0/1899014328 conn(0x7f27dc072020 msgr2=0x7f27dc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 -- 172.21.15.124:0/1899014328 shutdown_connections 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.555+0000 7f27e2978640 1 -- 172.21.15.124:0/1899014328 wait complete. 2023-12-13T23:55:57.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.556+0000 7f27e2978640 1 Processor -- start 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.556+0000 7f27e2978640 1 -- start start 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27dc0d2910 0x7f27dc0c8ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f27dc15d330 0x7f27dc0cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0c7bf0 con 0x7f27dc0d3900 2023-12-13T23:55:57.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0c7e40 con 0x7f27dc0d2910 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e2978640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27dc0c8120 con 0x7f27dc15d330 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43624/0 (socket says 172.21.15.124:43624) 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 -- 172.21.15.124:0/3543807299 learned_addr learned my addr 172.21.15.124:0/3543807299 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27dbfff640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27dc0d2910 0x7f27dc0c8ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27e0f15640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f27dc15d330 0x7f27dc0cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 -- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f27dc15d330 msgr2=0x7f27dc0cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f27dc15d330 0x7f27dc0cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 -- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27dc0d2910 msgr2=0x7f27dc0c8ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27dc0d2910 0x7f27dc0c8ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.557+0000 7f27db7fe640 1 -- 172.21.15.124:0/3543807299 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0094d0 con 0x7f27dc0d3900 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27db7fe640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f27d000c4d0 tx=0x7f27d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d00074e0 con 0x7f27dc0d3900 2023-12-13T23:55:57.563 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d0007660 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27dc0c8400 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d0004a20 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.558+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27dc0c89c0 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.559+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27dc0c8be0 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.559+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f27d0004b80 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.559+0000 7f27d8ff9640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 0x7f27c806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.560+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f27d0089020 con 0x7f27dc0d3900 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.560+0000 7f27dbfff640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 0x7f27c806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.560+0000 7f27dbfff640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 0x7f27c806d680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27cc036000 tx=0x7f27cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.564 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.564+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f27d0053cd0 con 0x7f27dc0d3900 2023-12-13T23:55:57.728 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.730+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f27dc069400 con 0x7f27dc0d3900 2023-12-13T23:55:57.729 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.730+0000 7f27d8ff9640 1 -- 172.21.15.124:0/3543807299 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7f27d00180a0 con 0x7f27dc0d3900 2023-12-13T23:55:57.729 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:57.730 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 msgr2=0x7f27c806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 0x7f27c806d680 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f27cc036000 tx=0x7f27cc008030 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 msgr2=0x7f27dc0ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f27d000c4d0 tx=0x7f27d000c990 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 shutdown_connections 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f27c806b1c0 0x7f27c806d680 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f27dc15d330 0x7f27dc0cd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f27dc0d3900 0x7f27dc0ccd90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 --2- 172.21.15.124:0/3543807299 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f27dc0d2910 0x7f27dc0c8ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 >> 172.21.15.124:0/3543807299 conn(0x7f27dc072020 msgr2=0x7f27dc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 shutdown_connections 2023-12-13T23:55:57.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.732+0000 7f27e2978640 1 -- 172.21.15.124:0/3543807299 wait complete. 2023-12-13T23:55:57.741 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:55:57.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.845+0000 7f358250c640 1 Processor -- start 2023-12-13T23:55:57.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 -- start start 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f357c15d330 0x7f357c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f357c0d3900 0x7f357c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357c0b2b10 con 0x7f357c0d2910 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357c0d1590 con 0x7f357c15d330 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f358250c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357c0d1870 con 0x7f357c0d3900 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f357bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f357c15d330 0x7f357c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f357bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f357c15d330 0x7f357c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43626/0 (socket says 172.21.15.124:43626) 2023-12-13T23:55:57.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.846+0000 7f357bfff640 1 -- 172.21.15.124:0/2934279731 learned_addr learned my addr 172.21.15.124:0/2934279731 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f357bfff640 1 -- 172.21.15.124:0/2934279731 >> v2:172.21.15.203:3300/0 conn(0x7f357c0d3900 msgr2=0x7f357c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f3580aa9640 1 --2- 172.21.15.124:0/2934279731 >> v2:172.21.15.203:3300/0 conn(0x7f357c0d3900 0x7f357c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f357bfff640 1 --2- 172.21.15.124:0/2934279731 >> v2:172.21.15.203:3300/0 conn(0x7f357c0d3900 0x7f357c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f357bfff640 1 -- 172.21.15.124:0/2934279731 >> v1:172.21.15.203:6789/0 conn(0x7f357c0d2910 legacy=0x7f357c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f357bfff640 1 -- 172.21.15.124:0/2934279731 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c0d1b50 con 0x7f357c15d330 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.847+0000 7f357bfff640 1 --2- 172.21.15.124:0/2934279731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c0b8b80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f357c06c500 tx=0x7f356c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567cd3efe08a72b0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f357affd640 1 -- 172.21.15.124:0/2934279731 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c0317e0 con 0x7f357c15d330 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f357affd640 1 -- 172.21.15.124:0/2934279731 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f356c031940 con 0x7f357c15d330 2023-12-13T23:55:57.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f357affd640 1 -- 172.21.15.124:0/2934279731 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f356c031cd0 con 0x7f357c15d330 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 -- 172.21.15.124:0/2934279731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 msgr2=0x7f357c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 --2- 172.21.15.124:0/2934279731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c0b8b80 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f357c06c500 tx=0x7f356c02b880 comp rx=0 tx=0).stop 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 -- 172.21.15.124:0/2934279731 shutdown_connections 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 --2- 172.21.15.124:0/2934279731 >> v2:172.21.15.203:3300/0 conn(0x7f357c0d3900 0x7f357c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 --2- 172.21.15.124:0/2934279731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c0b8b80 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 -- 172.21.15.124:0/2934279731 >> 172.21.15.124:0/2934279731 conn(0x7f357c072020 msgr2=0x7f357c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 -- 172.21.15.124:0/2934279731 shutdown_connections 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.848+0000 7f358250c640 1 -- 172.21.15.124:0/2934279731 wait complete. 2023-12-13T23:55:57.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.849+0000 7f358250c640 1 Processor -- start 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 -- start start 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 0x7f357c160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f357c0d3900 0x7f357c15e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c1609a0 con 0x7f357c15d330 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f357c15eaf0 con 0x7f357c0d3900 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f358250c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357c1598e0 con 0x7f357c0d2910 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f3580aa9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f3580aa9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43638/0 (socket says 172.21.15.124:43638) 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.850+0000 7f3580aa9640 1 -- 172.21.15.124:0/2899150981 learned_addr learned my addr 172.21.15.124:0/2899150981 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f357b7fe640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f357c0d3900 0x7f357c15e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f357bfff640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 0x7f357c160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 -- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 msgr2=0x7f357c160460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 0x7f357c160460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 -- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f357c0d3900 msgr2=0x7f357c15e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f357c0d3900 0x7f357c15e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 -- 172.21.15.124:0/2899150981 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0094d0 con 0x7f357c15d330 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f357bfff640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 0x7f357c160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3580aa9640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f357400cd40 tx=0x7f357400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574003b90 con 0x7f357c15d330 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3574003cf0 con 0x7f357c15d330 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357c1f3ad0 con 0x7f357c15d330 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574004240 con 0x7f357c15d330 2023-12-13T23:55:57.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.851+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357c0c7ed0 con 0x7f357c15d330 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.852+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357c069400 con 0x7f357c15d330 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.852+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f35740043a0 con 0x7f357c15d330 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.853+0000 7f3578ff9640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 0x7f356806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.853+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3574012070 con 0x7f357c15d330 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.853+0000 7f357bfff640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 0x7f356806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:57.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.854+0000 7f357bfff640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 0x7f356806d680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:57.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:57.857+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3574055120 con 0x7f357c15d330 2023-12-13T23:55:58.035 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.036+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f357c0d1060 con 0x7f357c15d330 2023-12-13T23:55:58.035 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.036+0000 7f3578ff9640 1 -- 172.21.15.124:0/2899150981 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3574058740 con 0x7f357c15d330 2023-12-13T23:55:58.035 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:55:58.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 msgr2=0x7f356806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 0x7f356806d680 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f356c036000 tx=0x7f356c03d040 comp rx=0 tx=0).stop 2023-12-13T23:55:58.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 msgr2=0x7f357c1593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f357400cd40 tx=0x7f357400a660 comp rx=0 tx=0).stop 2023-12-13T23:55:58.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 shutdown_connections 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f356806b1c0 0x7f356806d680 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f357c0d3900 0x7f357c15e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f357c15d330 0x7f357c1593a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 --2- 172.21.15.124:0/2899150981 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f357c0d2910 0x7f357c160460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 >> 172.21.15.124:0/2899150981 conn(0x7f357c072020 msgr2=0x7f357c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.038+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 shutdown_connections 2023-12-13T23:55:58.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.039+0000 7f358250c640 1 -- 172.21.15.124:0/2899150981 wait complete. 2023-12-13T23:55:58.048 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi203.front.sepia.ceph.com... 2023-12-13T23:55:58.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:58.048 DEBUG:teuthology.orchestra.run.smithi203:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2023-12-13T23:55:58.080 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:55:58.080 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:55:58.081 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:55:58.081 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-13T23:55:58.137 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-13T23:55:58.138 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:58.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.238+0000 7fc71d939640 1 Processor -- start 2023-12-13T23:55:58.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.238+0000 7fc71d939640 1 -- start start 2023-12-13T23:55:58.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc71d939640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc71815d910 0x7fc7180ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc71d939640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7180b49f0 con 0x7fc718150940 2023-12-13T23:55:58.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc71d939640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7180b3560 con 0x7fc718150560 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc71d939640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7180b3de0 con 0x7fc71815d910 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc7167fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc718150940 0x7fc71815d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40868/0 (socket says 172.21.15.124:40868) 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc7167fc640 1 -- 172.21.15.124:0/1328402232 learned_addr learned my addr 172.21.15.124:0/1328402232 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc7177fe640 1 --2- 172.21.15.124:0/1328402232 >> v2:172.21.15.203:3300/0 conn(0x7fc71815d910 0x7fc7180ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093223836 0 0) 0x7fc7180b49f0 con 0x7fc718150940 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.239+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc700003520 con 0x7fc718150940 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270588029 0 0) 0x7fc7180b3560 con 0x7fc718150560 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7180b49f0 con 0x7fc718150560 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4237445530 0 0) 0x7fc700003520 con 0x7fc718150940 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc7177fe640 1 -- 172.21.15.124:0/1328402232 >> v1:172.21.15.203:6789/0 conn(0x7fc718150560 legacy=0x7fc7180b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc7177fe640 1 -- 172.21.15.124:0/1328402232 >> v1:172.21.15.124:6789/0 conn(0x7fc718150940 legacy=0x7fc71815d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.240+0000 7fc7177fe640 1 -- 172.21.15.124:0/1328402232 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7180b40c0 con 0x7fc71815d910 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc7177fe640 1 --2- 172.21.15.124:0/1328402232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc7180ce0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7100097a0 tx=0x7fc71002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51376c3fd66b6468 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71002f050 con 0x7fc71815d910 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710031500 con 0x7fc71815d910 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc715ffb640 1 -- 172.21.15.124:0/1328402232 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70c001670 con 0x7fc71815d910 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 -- 172.21.15.124:0/1328402232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 msgr2=0x7fc7180ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 --2- 172.21.15.124:0/1328402232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc7180ce0e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc7100097a0 tx=0x7fc71002b820 comp rx=0 tx=0).stop 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 -- 172.21.15.124:0/1328402232 shutdown_connections 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 --2- 172.21.15.124:0/1328402232 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc7180ce0e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 -- 172.21.15.124:0/1328402232 >> 172.21.15.124:0/1328402232 conn(0x7fc71806a4d0 msgr2=0x7fc71806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 -- 172.21.15.124:0/1328402232 shutdown_connections 2023-12-13T23:55:58.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.241+0000 7fc71d939640 1 -- 172.21.15.124:0/1328402232 wait complete. 2023-12-13T23:55:58.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.242+0000 7fc71d939640 1 Processor -- start 2023-12-13T23:55:58.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 -- start start 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc718150560 0x7fc718158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc71814df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71814e940 con 0x7fc71814e4a0 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71814ec20 con 0x7fc71815d910 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc71d939640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71814ef00 con 0x7fc718150560 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc7177fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc7177fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43640/0 (socket says 172.21.15.124:43640) 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.243+0000 7fc7177fe640 1 -- 172.21.15.124:0/4206723032 learned_addr learned my addr 172.21.15.124:0/4206723032 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc716ffd640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc718150560 0x7fc718158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7167fc640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc71814df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 -- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc718150560 msgr2=0x7fc718158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc718150560 0x7fc718158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 -- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 msgr2=0x7fc71814df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc71814df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 -- 172.21.15.124:0/4206723032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710009400 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7167fc640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc71814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc7177fe640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc710036040 tx=0x7fc7100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710004230 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc710004390 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71814f1e0 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc710003850 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.244+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71814cb50 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.245+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc718069400 con 0x7fc71814e4a0 2023-12-13T23:55:58.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.245+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc71003a040 con 0x7fc71814e4a0 2023-12-13T23:55:58.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.246+0000 7fc6fffff640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 0x7fc70406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.246+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc7100aafc0 con 0x7fc71814e4a0 2023-12-13T23:55:58.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.246+0000 7fc716ffd640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 0x7fc70406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.247+0000 7fc716ffd640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 0x7fc70406d680 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc708001560 tx=0x7fc708009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.250+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc7100dde90 con 0x7fc71814e4a0 2023-12-13T23:55:58.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.451+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7181575c0 con 0x7fc71814e4a0 2023-12-13T23:55:58.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.451+0000 7fc6fffff640 1 -- 172.21.15.124:0/4206723032 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fc710038020 con 0x7fc71814e4a0 2023-12-13T23:55:58.450 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:58.450 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:58.450 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 msgr2=0x7fc70406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 0x7fc70406d680 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc708001560 tx=0x7fc708009040 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 msgr2=0x7fc71813bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc710036040 tx=0x7fc7100046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 shutdown_connections 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc70406b1c0 0x7fc70406d680 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc71815d910 0x7fc71814df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc71814e4a0 0x7fc71813bf80 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 --2- 172.21.15.124:0/4206723032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc718150560 0x7fc718158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 >> 172.21.15.124:0/4206723032 conn(0x7fc71806a4d0 msgr2=0x7fc7180ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 shutdown_connections 2023-12-13T23:55:58.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.453+0000 7fc71d939640 1 -- 172.21.15.124:0/4206723032 wait complete. 2023-12-13T23:55:58.463 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:55:58.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 Processor -- start 2023-12-13T23:55:58.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 -- start start 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa9b015d330 0x7fa9b00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9b00d3900 0x7fa9b00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b00b2b10 con 0x7fa9b00d2910 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00d1590 con 0x7fa9b00d3900 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9b4c3d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b00d1870 con 0x7fa9b015d330 2023-12-13T23:55:58.566 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.564+0000 7fa9aed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9b00d3900 0x7fa9b00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9b00d3900 0x7fa9b00b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43656/0 (socket says 172.21.15.124:43656) 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 -- 172.21.15.124:0/2906759620 learned_addr learned my addr 172.21.15.124:0/2906759620 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9ae59c640 1 --2- 172.21.15.124:0/2906759620 >> v2:172.21.15.203:3300/0 conn(0x7fa9b015d330 0x7fa9b00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 -- 172.21.15.124:0/2906759620 >> v2:172.21.15.203:3300/0 conn(0x7fa9b015d330 msgr2=0x7fa9b00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 --2- 172.21.15.124:0/2906759620 >> v2:172.21.15.203:3300/0 conn(0x7fa9b015d330 0x7fa9b00b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 -- 172.21.15.124:0/2906759620 >> v1:172.21.15.203:6789/0 conn(0x7fa9b00d2910 legacy=0x7fa9b00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.565+0000 7fa9aed9d640 1 -- 172.21.15.124:0/2906759620 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b00d1b50 con 0x7fa9b00d3900 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9aed9d640 1 --2- 172.21.15.124:0/2906759620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b00b6390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00d0d10 tx=0x7fa9a802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf71ecd8a347b6d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9add9b640 1 -- 172.21.15.124:0/2906759620 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a802c9f0 con 0x7fa9b00d3900 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9add9b640 1 -- 172.21.15.124:0/2906759620 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a802cb50 con 0x7fa9b00d3900 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9add9b640 1 -- 172.21.15.124:0/2906759620 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80343f0 con 0x7fa9b00d3900 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/2906759620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 msgr2=0x7fa9b00b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.567 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/2906759620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b00b6390 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fa9b00d0d10 tx=0x7fa9a802be50 comp rx=0 tx=0).stop 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/2906759620 shutdown_connections 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/2906759620 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b00b6390 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/2906759620 >> v2:172.21.15.203:3300/0 conn(0x7fa9b015d330 0x7fa9b00b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/2906759620 >> 172.21.15.124:0/2906759620 conn(0x7fa9b0072020 msgr2=0x7fa9b0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/2906759620 shutdown_connections 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.566+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/2906759620 wait complete. 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 Processor -- start 2023-12-13T23:55:58.568 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 -- start start 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9b00d2910 0x7fa9b0154350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9b015d330 0x7fa9b01601e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b00cc5d0 con 0x7fa9b00d3900 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b0154890 con 0x7fa9b00d2910 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9b4c3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b015d980 con 0x7fa9b015d330 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9a7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9a7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43660/0 (socket says 172.21.15.124:43660) 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.568+0000 7fa9a7fff640 1 -- 172.21.15.124:0/3511431785 learned_addr learned my addr 172.21.15.124:0/3511431785 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9ae59c640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9b00d2910 0x7fa9b0154350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9aed9d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9b015d330 0x7fa9b01601e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 -- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9b015d330 msgr2=0x7fa9b01601e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9b015d330 0x7fa9b01601e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 -- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9b00d2910 msgr2=0x7fa9b0154350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9b00d2910 0x7fa9b0154350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 -- 172.21.15.124:0/3511431785 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a8009b30 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a7fff640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa99800c430 tx=0x7fa99800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011790 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9980118f0 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b015dc60 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa998011ca0 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.569+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b015e220 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.570+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b0069400 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.571+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa99801c460 con 0x7fa9b00d3900 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.571+0000 7fa9a6ffd640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 0x7fa99006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.571+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa998089880 con 0x7fa9b00d3900 2023-12-13T23:55:58.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.571+0000 7fa9ae59c640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 0x7fa99006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.572+0000 7fa9ae59c640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 0x7fa99006d6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00079a0 tx=0x7fa9a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.575+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa9980545e0 con 0x7fa9b00d3900 2023-12-13T23:55:58.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.744+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9b0159ef0 con 0x7fa9b00d3900 2023-12-13T23:55:58.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.745+0000 7fa9a6ffd640 1 -- 172.21.15.124:0/3511431785 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16815 (secure 0 0 0) 0x7fa998057c00 con 0x7fa9b00d3900 2023-12-13T23:55:58.744 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:58.744 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":23,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:53.877550+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 msgr2=0x7fa99006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 0x7fa99006d6b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa9a00079a0 tx=0x7fa9a0008040 comp rx=0 tx=0).stop 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 msgr2=0x7fa9b015ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa99800c430 tx=0x7fa99800c8f0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 shutdown_connections 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa99006b1f0 0x7fa99006d6b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa9b015d330 0x7fa9b01601e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9b00d3900 0x7fa9b015ed00 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 --2- 172.21.15.124:0/3511431785 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa9b00d2910 0x7fa9b0154350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 >> 172.21.15.124:0/3511431785 conn(0x7fa9b0072020 msgr2=0x7fa9b0072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 shutdown_connections 2023-12-13T23:55:58.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.747+0000 7fa9b4c3d640 1 -- 172.21.15.124:0/3511431785 wait complete. 2023-12-13T23:55:58.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:55:10.129448+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:55:11.064925+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:55:58.757 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:55:58.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.861+0000 7f03a0d5a640 1 Processor -- start 2023-12-13T23:55:58.860 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.861+0000 7f03a0d5a640 1 -- start start 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f03a0d5a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f039c0b1400 0x7f039c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f03a0d5a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c15e730 0x7f039c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f03a0d5a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f039c15e2a0 con 0x7f039c143490 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f03a0d5a640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039c15b520 con 0x7f039c0b1400 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f03a0d5a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c143870 con 0x7f039c15e730 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f039c0b1400 0x7f039c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f039c0b1400 0x7f039c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43674/0 (socket says 172.21.15.124:43674) 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 -- 172.21.15.124:0/3194337507 learned_addr learned my addr 172.21.15.124:0/3194337507 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 -- 172.21.15.124:0/3194337507 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c15e730 msgr2=0x7f039c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 --2- 172.21.15.124:0/3194337507 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c15e730 0x7f039c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 -- 172.21.15.124:0/3194337507 >> v1:172.21.15.203:6789/0 conn(0x7f039c143490 legacy=0x7f039c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:58.862 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.862+0000 7f0393fff640 1 -- 172.21.15.124:0/3194337507 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c15b240 con 0x7f039c0b1400 2023-12-13T23:55:58.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f0393fff640 1 --2- 172.21.15.124:0/3194337507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c0b1400 0x7f039c15a4e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f038400bb10 tx=0x7f038402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf71f3e9a6c41135 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f0399d9b640 1 -- 172.21.15.124:0/3194337507 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038402ee70 con 0x7f039c0b1400 2023-12-13T23:55:58.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f0399d9b640 1 -- 172.21.15.124:0/3194337507 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0384036440 con 0x7f039c0b1400 2023-12-13T23:55:58.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f0399d9b640 1 -- 172.21.15.124:0/3194337507 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03840367d0 con 0x7f039c0b1400 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 -- 172.21.15.124:0/3194337507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c0b1400 msgr2=0x7f039c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/3194337507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c0b1400 0x7f039c15a4e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f038400bb10 tx=0x7f038402e0c0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 -- 172.21.15.124:0/3194337507 shutdown_connections 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/3194337507 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c15e730 0x7f039c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/3194337507 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c0b1400 0x7f039c15a4e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 -- 172.21.15.124:0/3194337507 >> 172.21.15.124:0/3194337507 conn(0x7f039c06a4d0 msgr2=0x7f039c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 -- 172.21.15.124:0/3194337507 shutdown_connections 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.863+0000 7f03a0d5a640 1 -- 172.21.15.124:0/3194337507 wait complete. 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 Processor -- start 2023-12-13T23:55:58.864 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 -- start start 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c0b1400 0x7f039c13ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f039c15e730 0x7f039c1ef130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c15d540 con 0x7f039c143490 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c159f30 con 0x7f039c15e730 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.865+0000 7f03a0d5a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f039c13ffc0 con 0x7f039c0b1400 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43690/0 (socket says 172.21.15.124:43690) 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 -- 172.21.15.124:0/1512532572 learned_addr learned my addr 172.21.15.124:0/1512532572 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f039a59c640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c0b1400 0x7f039c13ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f039ad9d640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f039c15e730 0x7f039c1ef130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 -- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c0b1400 msgr2=0x7f039c13ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c0b1400 0x7f039c13ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 -- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f039c15e730 msgr2=0x7f039c1ef130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f039c15e730 0x7f039c1ef130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 -- 172.21.15.124:0/1512532572 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038400b770 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0393fff640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f038400b2c0 tx=0x7f03840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0384034950 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0384034ad0 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0384034e80 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c13e460 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.866+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c140540 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.867+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f039c069400 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.868+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f038403d040 con 0x7f039c143490 2023-12-13T23:55:58.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.868+0000 7f0392ffd640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 0x7f037c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:58.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.868+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f03840ae5a0 con 0x7f039c143490 2023-12-13T23:55:58.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.868+0000 7f039a59c640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 0x7f037c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:58.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.869+0000 7f039a59c640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 0x7f037c06d680 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f038c007fd0 tx=0x7f038c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:58.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:58.872+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0384079250 con 0x7f039c143490 2023-12-13T23:55:59.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.042+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f039c0b4880 con 0x7f039c143490 2023-12-13T23:55:59.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.043+0000 7f0392ffd640 1 -- 172.21.15.124:0/1512532572 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v23) v1 ==== 93+0+10 (secure 0 0 0) 0x7f038407c870 con 0x7f039c143490 2023-12-13T23:55:59.041 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:55:59.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 msgr2=0x7f037c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 0x7f037c06d680 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f038c007fd0 tx=0x7f038c009040 comp rx=0 tx=0).stop 2023-12-13T23:55:59.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 msgr2=0x7f039c1599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f038400b2c0 tx=0x7f03840046c0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 shutdown_connections 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f037c06b1c0 0x7f037c06d680 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f039c15e730 0x7f039c1ef130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f039c143490 0x7f039c1599f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 --2- 172.21.15.124:0/1512532572 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f039c0b1400 0x7f039c13ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.044+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 >> 172.21.15.124:0/1512532572 conn(0x7f039c06a4d0 msgr2=0x7f039c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.045+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 shutdown_connections 2023-12-13T23:55:59.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.045+0000 7f03a0d5a640 1 -- 172.21.15.124:0/1512532572 wait complete. 2023-12-13T23:55:59.054 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:55:59.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 Processor -- start 2023-12-13T23:55:59.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 -- start start 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f95c015d2d0 0x7f95c00b8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c01614e0 con 0x7f95c00ce650 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95c01617c0 con 0x7f95c0160f70 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.158+0000 7f95c5553640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95c0161aa0 con 0x7f95c015d2d0 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f95c015d2d0 0x7f95c00b8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f95c015d2d0 0x7f95c00b8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43700/0 (socket says 172.21.15.124:43700) 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95bf7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f95c00ce650 0x7f95c00d1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40876/0 (socket says 172.21.15.124:40876) 2023-12-13T23:55:59.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 -- 172.21.15.124:0/3860485361 learned_addr learned my addr 172.21.15.124:0/3860485361 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 -- 172.21.15.124:0/3860485361 >> v1:172.21.15.203:6789/0 conn(0x7f95c0160f70 legacy=0x7f95c00cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 -- 172.21.15.124:0/3860485361 >> v1:172.21.15.124:6789/0 conn(0x7f95c00ce650 legacy=0x7f95c00d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95bdffb640 1 -- 172.21.15.124:0/3860485361 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710823938 0 0) 0x7f95c01614e0 con 0x7f95c00ce650 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.159+0000 7f95beffd640 1 -- 172.21.15.124:0/3860485361 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c0161d80 con 0x7f95c015d2d0 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95beffd640 1 --2- 172.21.15.124:0/3860485361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b8b90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009870 tx=0x7f95b002ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d7f0102446a00a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95bdffb640 1 -- 172.21.15.124:0/3860485361 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b002f840 con 0x7f95c015d2d0 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95bdffb640 1 -- 172.21.15.124:0/3860485361 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b002f9a0 con 0x7f95c015d2d0 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95bdffb640 1 -- 172.21.15.124:0/3860485361 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b002fd30 con 0x7f95c015d2d0 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 -- 172.21.15.124:0/3860485361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 msgr2=0x7f95c00b8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 --2- 172.21.15.124:0/3860485361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b8b90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009870 tx=0x7f95b002ba20 comp rx=0 tx=0).stop 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 -- 172.21.15.124:0/3860485361 shutdown_connections 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 --2- 172.21.15.124:0/3860485361 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b8b90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 -- 172.21.15.124:0/3860485361 >> 172.21.15.124:0/3860485361 conn(0x7f95c0072230 msgr2=0x7f95c00cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 -- 172.21.15.124:0/3860485361 shutdown_connections 2023-12-13T23:55:59.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.160+0000 7f95c5553640 1 -- 172.21.15.124:0/3860485361 wait complete. 2023-12-13T23:55:59.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 Processor -- start 2023-12-13T23:55:59.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 -- start start 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f95c00ce650 0x7f95c00ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c0160f70 0x7f95c00cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c0155e50 con 0x7f95c015d2d0 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00cbe90 con 0x7f95c0160f70 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95c5553640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95c00ca8b0 con 0x7f95c00ce650 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95be7fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95be7fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43706/0 (socket says 172.21.15.124:43706) 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.162+0000 7f95be7fc640 1 -- 172.21.15.124:0/3684986633 learned_addr learned my addr 172.21.15.124:0/3684986633 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 -- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f95c00ce650 msgr2=0x7f95c00ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f95c00ce650 0x7f95c00ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 -- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c0160f70 msgr2=0x7f95c00cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c0160f70 0x7f95c00cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 -- 172.21.15.124:0/3684986633 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b00094d0 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95be7fc640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c400 tx=0x7f95b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b40052d0 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b4005430 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c00cab90 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b4003830 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.163+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c00cb150 con 0x7f95c015d2d0 2023-12-13T23:55:59.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.164+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c0069400 con 0x7f95c015d2d0 2023-12-13T23:55:59.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.164+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f95b4003990 con 0x7f95c015d2d0 2023-12-13T23:55:59.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.165+0000 7f95a77fe640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 0x7f95ac06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.165+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f95b4088f70 con 0x7f95c015d2d0 2023-12-13T23:55:59.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.165+0000 7f95beffd640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 0x7f95ac06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.166+0000 7f95beffd640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 0x7f95ac06d6b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f95b0036000 tx=0x7f95b0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.169+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f95b40bbe90 con 0x7f95c015d2d0 2023-12-13T23:55:59.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.335+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f95c00d0aa0 con 0x7f95c015d2d0 2023-12-13T23:55:59.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.336+0000 7f95a77fe640 1 -- 172.21.15.124:0/3684986633 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f95b4053cb0 con 0x7f95c015d2d0 2023-12-13T23:55:59.334 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 msgr2=0x7f95ac06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 0x7f95ac06d6b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f95b0036000 tx=0x7f95b0030000 comp rx=0 tx=0).stop 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 msgr2=0x7f95c00b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f95b400c400 tx=0x7f95b400c8c0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 shutdown_connections 2023-12-13T23:55:59.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f95ac06b1f0 0x7f95ac06d6b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f95c0160f70 0x7f95c00cb950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f95c015d2d0 0x7f95c00b56f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 --2- 172.21.15.124:0/3684986633 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f95c00ce650 0x7f95c00ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 >> 172.21.15.124:0/3684986633 conn(0x7f95c0072230 msgr2=0x7f95c00cdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 shutdown_connections 2023-12-13T23:55:59.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.337+0000 7f95c5553640 1 -- 172.21.15.124:0/3684986633 wait complete. 2023-12-13T23:55:59.347 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:55:59.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.451+0000 7fa4ad856640 1 Processor -- start 2023-12-13T23:55:59.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.451+0000 7fa4ad856640 1 -- start start 2023-12-13T23:55:59.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4ad856640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa4a80d2910 0x7fa4a80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4ad856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a80d3730 0x7fa4a80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4ad856640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a80b2b10 con 0x7fa4a815d330 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4ad856640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4a80d1590 con 0x7fa4a80d2910 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4ad856640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80d1870 con 0x7fa4a80d3730 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a6ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa4a815d330 0x7fa4a80b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40890/0 (socket says 172.21.15.124:40890) 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a6ffd640 1 -- 172.21.15.124:0/197650234 learned_addr learned my addr 172.21.15.124:0/197650234 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268672986 0 0) 0x7fa4a80b2b10 con 0x7fa4a815d330 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa494003520 con 0x7fa4a815d330 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa49ffff640 1 --2- 172.21.15.124:0/197650234 >> v2:172.21.15.203:3300/0 conn(0x7fa4a80d2910 0x7fa4a80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a80d3730 0x7fa4a80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4199523549 0 0) 0x7fa494003520 con 0x7fa4a815d330 2023-12-13T23:55:59.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a80d3730 msgr2=0x7fa4a80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a80d3730 0x7fa4a80b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 >> v2:172.21.15.203:3300/0 conn(0x7fa4a80d2910 msgr2=0x7fa4a80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 --2- 172.21.15.124:0/197650234 >> v2:172.21.15.203:3300/0 conn(0x7fa4a80d2910 0x7fa4a80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80d3b90 con 0x7fa4a815d330 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fa498001670 con 0x7fa4a815d330 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 >> v1:172.21.15.124:6789/0 conn(0x7fa4a815d330 legacy=0x7fa4a80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa494003970 0x7fa494003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa494004360 0x7fa494004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494005000 0x7fa49400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.452+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80d1590 con 0x7fa494004360 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80d1870 con 0x7fa494005000 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49400bfa0 con 0x7fa494003970 2023-12-13T23:55:59.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a80d3730 0x7fa4a80b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa494004360 0x7fa494004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa4a6ffd640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa494003970 0x7fa494003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494005000 0x7fa49400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 -- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa494003970 msgr2=0x7fa494003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa494003970 0x7fa494003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 -- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494005000 msgr2=0x7fa49400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa494005000 0x7fa49400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.453+0000 7fa49ffff640 1 -- 172.21.15.124:0/197650234 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa494003520 con 0x7fa494004360 2023-12-13T23:55:59.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa49ffff640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa494004360 0x7fa494004840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa490009a90 tx=0x7fa4940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc57a887ccb1043 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49002c6b0 con 0x7fa494004360 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa49002c810 con 0x7fa494004360 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4a67fc640 1 -- 172.21.15.124:0/197650234 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa49002cba0 con 0x7fa494004360 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 -- 172.21.15.124:0/197650234 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa494004360 msgr2=0x7fa494004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 --2- 172.21.15.124:0/197650234 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa494004360 0x7fa494004840 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fa490009a90 tx=0x7fa4940057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4a77fe640 1 -- 172.21.15.124:0/197650234 reap_dead start 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 -- 172.21.15.124:0/197650234 shutdown_connections 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 -- 172.21.15.124:0/197650234 >> 172.21.15.124:0/197650234 conn(0x7fa4a8072020 msgr2=0x7fa4a8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 -- 172.21.15.124:0/197650234 shutdown_connections 2023-12-13T23:55:59.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.454+0000 7fa4ad856640 1 -- 172.21.15.124:0/197650234 wait complete. 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 Processor -- start 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 -- start start 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a81603a0 0x7fa4a80ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a8160780 0x7fa4a80cbe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a81459c0 con 0x7fa4a8144fd0 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80cc7c0 con 0x7fa4a8160780 2023-12-13T23:55:59.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4ad856640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a80ccaa0 con 0x7fa4a81603a0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4a77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4a77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43736/0 (socket says 172.21.15.124:43736) 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.456+0000 7fa4a77fe640 1 -- 172.21.15.124:0/498824089 learned_addr learned my addr 172.21.15.124:0/498824089 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa49ffff640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a8160780 0x7fa4a80cbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a6ffd640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a81603a0 0x7fa4a80ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 -- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a81603a0 msgr2=0x7fa4a80ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a81603a0 0x7fa4a80ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 -- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a8160780 msgr2=0x7fa4a80cbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a8160780 0x7fa4a80cbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 -- 172.21.15.124:0/498824089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a80ccd80 con 0x7fa4a8144fd0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a77fe640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c520 tx=0x7fa4a0009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0007d30 con 0x7fa4a8144fd0 2023-12-13T23:55:59.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4a000f050 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a80cd060 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0014630 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.457+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a80cc3d0 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.458+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a80cd580 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.458+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa4a0014790 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.459+0000 7fa4a4ff9640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 0x7fa48806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.459+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa4a0089f50 con 0x7fa4a8144fd0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.459+0000 7fa4a6ffd640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 0x7fa48806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.460+0000 7fa4a6ffd640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 0x7fa48806d6b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa498004840 tx=0x7fa498009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.463+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa4a80cd580 con 0x7fa4a8144fd0 2023-12-13T23:55:59.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.629+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa4a80cd580 con 0x7fa4a8144fd0 2023-12-13T23:55:59.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.629+0000 7fa4a4ff9640 1 -- 172.21.15.124:0/498824089 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa4a80cd580 con 0x7fa4a8144fd0 2023-12-13T23:55:59.628 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:55:59.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 msgr2=0x7fa48806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 0x7fa48806d6b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa498004840 tx=0x7fa498009040 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 msgr2=0x7fa4a8145480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fa4a806c520 tx=0x7fa4a0009f60 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 shutdown_connections 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa48806b1f0 0x7fa48806d6b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4a8160780 0x7fa4a80cbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4a8144fd0 0x7fa4a8145480 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 --2- 172.21.15.124:0/498824089 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4a81603a0 0x7fa4a80ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 >> 172.21.15.124:0/498824089 conn(0x7fa4a8072020 msgr2=0x7fa4a80b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 shutdown_connections 2023-12-13T23:55:59.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.631+0000 7fa4ad856640 1 -- 172.21.15.124:0/498824089 wait complete. 2023-12-13T23:55:59.640 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:55:59.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.740+0000 7fa0aca23640 1 Processor -- start 2023-12-13T23:55:59.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.740+0000 7fa0aca23640 1 -- start start 2023-12-13T23:55:59.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0aca23640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0a80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0aca23640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80b2b10 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0aca23640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80d3b50 con 0x7fa0a815d330 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0aca23640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80d1590 con 0x7fa0a80d2910 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0a6d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa0a80d3730 0x7fa0a80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40898/0 (socket says 172.21.15.124:40898) 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.741+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/907734747 learned_addr learned my addr 172.21.15.124:0/907734747 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a5d9b640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0a80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254363184 0 0) 0x7fa0a80b2b10 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa090003520 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290282185 0 0) 0x7fa090003520 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 msgr2=0x7fa0a80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0a80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 >> v1:172.21.15.203:6789/0 conn(0x7fa0a815d330 legacy=0x7fa0a80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80d1870 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fa0a0001670 con 0x7fa0a80d3730 2023-12-13T23:55:59.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 >> v1:172.21.15.124:6789/0 conn(0x7fa0a80d3730 legacy=0x7fa0a80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0900039a0 0x7fa090003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900047c0 0x7fa090004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80d1590 con 0x7fa0900039a0 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa09000bae0 con 0x7fa0900047c0 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa09000bca0 con 0x7fa0a80d2910 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0900039a0 0x7fa090003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a6d9d640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a5d9b640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900047c0 0x7fa090004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 -- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 msgr2=0x7fa0900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a80d2910 0x7fa0900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 -- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900047c0 msgr2=0x7fa090004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:55:59.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900047c0 0x7fa090004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a659c640 1 -- 172.21.15.124:0/907734747 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa090003520 con 0x7fa0900039a0 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.742+0000 7fa0a5d9b640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900047c0 0x7fa090004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.743+0000 7fa0a659c640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0900039a0 0x7fa090003dc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa098001e50 tx=0x7fa0900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0b394d5994ab1a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.743+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098003b70 con 0x7fa0900039a0 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.743+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098003cd0 con 0x7fa0900039a0 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.743+0000 7fa0a559a640 1 -- 172.21.15.124:0/907734747 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098005310 con 0x7fa0900039a0 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 -- 172.21.15.124:0/907734747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0900039a0 msgr2=0x7fa090003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 --2- 172.21.15.124:0/907734747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0900039a0 0x7fa090003dc0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fa098001e50 tx=0x7fa0900057f0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/907734747 reap_dead start 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 -- 172.21.15.124:0/907734747 shutdown_connections 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 -- 172.21.15.124:0/907734747 >> 172.21.15.124:0/907734747 conn(0x7fa0a8072020 msgr2=0x7fa0a8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 -- 172.21.15.124:0/907734747 shutdown_connections 2023-12-13T23:55:59.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.744+0000 7fa0aca23640 1 -- 172.21.15.124:0/907734747 wait complete. 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.745+0000 7fa0aca23640 1 Processor -- start 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.745+0000 7fa0aca23640 1 -- start start 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900039a0 0x7fa0a815eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a8143540 0x7fa0a815f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8152ea0 con 0x7fa0a81439a0 2023-12-13T23:55:59.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a815f340 con 0x7fa0900039a0 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0aca23640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8159ef0 con 0x7fa0a8143540 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43746/0 (socket says 172.21.15.124:43746) 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/3667356629 learned_addr learned my addr 172.21.15.124:0/3667356629 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a8143540 msgr2=0x7fa0a815f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a8143540 0x7fa0a815f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900039a0 msgr2=0x7fa0a815eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900039a0 0x7fa0a815eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.746+0000 7fa0a6d9d640 1 -- 172.21.15.124:0/3667356629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8156ca0 con 0x7fa0a81439a0 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa0a6d9d640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa0a808c510 tx=0x7fa0a0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0001640 con 0x7fa0a81439a0 2023-12-13T23:55:59.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a000f890 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a8156f80 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a000fc40 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.747+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8157260 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.748+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a8069400 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.748+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa0a001d460 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.748+0000 7fa096ffd640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 0x7fa08806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.749+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa0a0089960 con 0x7fa0a81439a0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.749+0000 7fa0a659c640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 0x7fa08806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:55:59.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.749+0000 7fa0a659c640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 0x7fa08806d6b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa098034000 tx=0x7fa098030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:55:59.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.753+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa0a00546a0 con 0x7fa0a81439a0 2023-12-13T23:55:59.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.953+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0a81602a0 con 0x7fa0a81439a0 2023-12-13T23:55:59.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.953+0000 7fa096ffd640 1 -- 172.21.15.124:0/3667356629 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7fa0a0057cc0 con 0x7fa0a81439a0 2023-12-13T23:55:59.952 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:55:59.952 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:55:59.953 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 msgr2=0x7fa08806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 0x7fa08806d6b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa098034000 tx=0x7fa098030000 comp rx=0 tx=0).stop 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 msgr2=0x7fa0a815fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa0a808c510 tx=0x7fa0a0007720 comp rx=0 tx=0).stop 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 shutdown_connections 2023-12-13T23:55:59.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa08806b1f0 0x7fa08806d6b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa0a81439a0 0x7fa0a815fd60 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa0a8143540 0x7fa0a815f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 --2- 172.21.15.124:0/3667356629 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa0900039a0 0x7fa0a815eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.955+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 >> 172.21.15.124:0/3667356629 conn(0x7fa0a8072020 msgr2=0x7fa0a80c7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.956+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 shutdown_connections 2023-12-13T23:55:59.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:55:59.956+0000 7fa0aca23640 1 -- 172.21.15.124:0/3667356629 wait complete. 2023-12-13T23:55:59.965 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:00.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 Processor -- start 2023-12-13T23:56:00.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 -- start start 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f9680071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968015f3a0 con 0x7f9680072130 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968015f680 con 0x7f96801448f0 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.066+0000 7f968723d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800714c0 con 0x7f968006a4d0 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f96857da640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f9680072130 0x7f968015b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40902/0 (socket says 172.21.15.124:40902) 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f96857da640 1 -- 172.21.15.124:0/144265159 learned_addr learned my addr 172.21.15.124:0/144265159 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952151220 0 0) 0x7f968015f3a0 con 0x7f9680072130 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f966c003520 con 0x7f9680072130 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967ffff640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f9680071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777921768 0 0) 0x7f966c003520 con 0x7f9680072130 2023-12-13T23:56:00.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 msgr2=0x7f9680071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f9680071bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 >> v1:172.21.15.203:6789/0 conn(0x7f96801448f0 legacy=0x7f968015ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968015f960 con 0x7f9680072130 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f9668001670 con 0x7f9680072130 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 >> v1:172.21.15.124:6789/0 conn(0x7f9680072130 legacy=0x7f968015b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f966c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f966c0047c0 0x7f966c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f966c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800714c0 con 0x7f966c0039a0 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966c0054b0 con 0x7f966c0047c0 2023-12-13T23:56:00.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966c005670 con 0x7f968006a4d0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f9684fd9640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f966c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f967ffff640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f966c0047c0 0x7f966c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f96857da640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f966c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.067+0000 7f9684fd9640 1 -- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 msgr2=0x7f966c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f9684fd9640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f968006a4d0 0x7f966c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f9684fd9640 1 -- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f966c0047c0 msgr2=0x7f966c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f9684fd9640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f966c0047c0 0x7f966c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f9684fd9640 1 -- 172.21.15.124:0/144265159 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966c003520 con 0x7f966c0039a0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f967ffff640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f966c0047c0 0x7f966c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f9684fd9640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f966c003dc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9678001e50 tx=0x7f966c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dacc8723e854b29 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678003b70 con 0x7f966c0039a0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678003cd0 con 0x7f966c0039a0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.068+0000 7f967f7fe640 1 -- 172.21.15.124:0/144265159 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678005310 con 0x7f966c0039a0 2023-12-13T23:56:00.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 -- 172.21.15.124:0/144265159 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 msgr2=0x7f966c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 --2- 172.21.15.124:0/144265159 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f966c003dc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9678001e50 tx=0x7f966c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f96857da640 1 -- 172.21.15.124:0/144265159 reap_dead start 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 -- 172.21.15.124:0/144265159 shutdown_connections 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 -- 172.21.15.124:0/144265159 >> 172.21.15.124:0/144265159 conn(0x7f96800b1de0 msgr2=0x7f96800b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 -- 172.21.15.124:0/144265159 shutdown_connections 2023-12-13T23:56:00.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.069+0000 7f968723d640 1 -- 172.21.15.124:0/144265159 wait complete. 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.070+0000 7f968723d640 1 Processor -- start 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.070+0000 7f968723d640 1 -- start start 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96800aa720 0x7f96800ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96800abca0 0x7f96801424a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800aac00 con 0x7f966c0039a0 2023-12-13T23:56:00.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96800aaee0 con 0x7f96800abca0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f968723d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96800ab1c0 con 0x7f96800aa720 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43762/0 (socket says 172.21.15.124:43762) 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 -- 172.21.15.124:0/2988565982 learned_addr learned my addr 172.21.15.124:0/2988565982 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 -- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96800aa720 msgr2=0x7f96800ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96800aa720 0x7f96800ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 -- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96800abca0 msgr2=0x7f96801424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96800abca0 0x7f96801424a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.071+0000 7f9684fd9640 1 -- 172.21.15.124:0/2988565982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96800ab4a0 con 0x7f966c0039a0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f9684fd9640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f80 tx=0x7f96780014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967803e070 con 0x7f966c0039a0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9678006b50 con 0x7f966c0039a0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96800ab780 con 0x7f966c0039a0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9678007030 con 0x7f966c0039a0 2023-12-13T23:56:00.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.072+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96800aed00 con 0x7f966c0039a0 2023-12-13T23:56:00.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.073+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96800afba0 con 0x7f966c0039a0 2023-12-13T23:56:00.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.073+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9678007190 con 0x7f966c0039a0 2023-12-13T23:56:00.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.074+0000 7f967d7fa640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 0x7f966006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.074+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f96780ab260 con 0x7f966c0039a0 2023-12-13T23:56:00.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.074+0000 7f967ffff640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 0x7f966006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.074+0000 7f967ffff640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 0x7f966006d6b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96800af110 tx=0x7f9670004160 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.078+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9678075f00 con 0x7f966c0039a0 2023-12-13T23:56:00.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.248+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9680067420 con 0x7f966c0039a0 2023-12-13T23:56:00.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.249+0000 7f967d7fa640 1 -- 172.21.15.124:0/2988565982 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16815 (secure 0 0 0) 0x7f9678079520 con 0x7f966c0039a0 2023-12-13T23:56:00.248 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:00.248 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":23,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:53.877550+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:00.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.251+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 msgr2=0x7f966006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.251+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 0x7f966006d6b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f96800af110 tx=0x7f9670004160 comp rx=0 tx=0).stop 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.251+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 msgr2=0x7f968013f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.251+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f9678000f80 tx=0x7f96780014a0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 shutdown_connections 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f966006b1f0 0x7f966006d6b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f96800abca0 0x7f96801424a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f96800aa720 0x7f96800ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 --2- 172.21.15.124:0/2988565982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f966c0039a0 0x7f968013f8f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 >> 172.21.15.124:0/2988565982 conn(0x7f96800b1de0 msgr2=0x7f96800b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 shutdown_connections 2023-12-13T23:56:00.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.252+0000 7f968723d640 1 -- 172.21.15.124:0/2988565982 wait complete. 2023-12-13T23:56:00.262 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-13T23:56:00.262 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:56:00.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.363+0000 7f5977768640 1 Processor -- start 2023-12-13T23:56:00.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 -- start start 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 0x7f59700b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f59700d2d90 0x7f59700b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59700b2b10 con 0x7f59700d2910 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59700d1590 con 0x7f59700d2d90 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.364+0000 7f5977768640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59700d1870 con 0x7f597015d330 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5974d03640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f59700d2910 0x7f59700d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:40918/0 (socket says 172.21.15.124:40918) 2023-12-13T23:56:00.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5974d03640 1 -- 172.21.15.124:0/3455976147 learned_addr learned my addr 172.21.15.124:0/3455976147 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975d05640 1 --2- 172.21.15.124:0/3455976147 >> v2:172.21.15.203:3300/0 conn(0x7f59700d2d90 0x7f59700b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975504640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 0x7f59700b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531257126 0 0) 0x7f59700b2b10 con 0x7f59700d2910 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f595c003520 con 0x7f59700d2910 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975d05640 1 -- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 msgr2=0x7f59700b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975d05640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 0x7f59700b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975d05640 1 -- 172.21.15.124:0/3455976147 >> v1:172.21.15.124:6789/0 conn(0x7f59700d2910 legacy=0x7f59700d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786389722 0 0) 0x7f595c003520 con 0x7f59700d2910 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975d05640 1 -- 172.21.15.124:0/3455976147 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59700d1b50 con 0x7f59700d2d90 2023-12-13T23:56:00.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.365+0000 7f5975504640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 0x7f59700b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:00.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.366+0000 7f5975d05640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2d90 0x7f59700b6390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f596802b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c9f0550c5ca940 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.366+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5960001670 con 0x7f59700d2d90 2023-12-13T23:56:00.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.366+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f596802e050 con 0x7f59700d2d90 2023-12-13T23:56:00.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.366+0000 7f5967fff640 1 -- 172.21.15.124:0/3455976147 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968030710 con 0x7f59700d2d90 2023-12-13T23:56:00.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 -- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2d90 msgr2=0x7f59700b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2d90 0x7f59700b6390 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f59680097a0 tx=0x7f596802b710 comp rx=0 tx=0).stop 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 -- 172.21.15.124:0/3455976147 shutdown_connections 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2d90 0x7f59700b6390 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 --2- 172.21.15.124:0/3455976147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f597015d330 0x7f59700b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 -- 172.21.15.124:0/3455976147 >> 172.21.15.124:0/3455976147 conn(0x7f5970072020 msgr2=0x7f5970072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 -- 172.21.15.124:0/3455976147 shutdown_connections 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.367+0000 7f5977768640 1 -- 172.21.15.124:0/3455976147 wait complete. 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 Processor -- start 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 -- start start 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2910 0x7f59701569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59700d2d90 0x7f59701f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59701f3cf0 con 0x7f597015d330 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5970156f20 con 0x7f59700d2910 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.368+0000 7f5977768640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59701f6bc0 con 0x7f59700d2d90 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43768/0 (socket says 172.21.15.124:43768) 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 -- 172.21.15.124:0/3309430943 learned_addr learned my addr 172.21.15.124:0/3309430943 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975504640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2910 0x7f59701569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5974d03640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59700d2d90 0x7f59701f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 -- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59700d2d90 msgr2=0x7f59701f6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59700d2d90 0x7f59701f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 -- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2910 msgr2=0x7f59701569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2910 0x7f59701569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 -- 172.21.15.124:0/3309430943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f596c009a70 con 0x7f597015d330 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5975d05640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f596802b750 tx=0x7f59680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968003ea0 con 0x7f597015d330 2023-12-13T23:56:00.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59680044c0 con 0x7f597015d330 2023-12-13T23:56:00.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5968030750 con 0x7f597015d330 2023-12-13T23:56:00.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5968009400 con 0x7f597015d330 2023-12-13T23:56:00.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.369+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59701f3f00 con 0x7f597015d330 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.370+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5970069400 con 0x7f597015d330 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.371+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5968039070 con 0x7f597015d330 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.371+0000 7f5965ffb640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 0x7f595406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.371+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f59680ab170 con 0x7f597015d330 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.371+0000 7f5975504640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 0x7f595406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.372+0000 7f5975504640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 0x7f595406d680 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f596c007600 tx=0x7f596c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.375+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5968075eb0 con 0x7f597015d330 2023-12-13T23:56:00.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.576+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f597015cd30 con 0x7f597015d330 2023-12-13T23:56:00.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.576+0000 7f5965ffb640 1 -- 172.21.15.124:0/3309430943 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f5968040090 con 0x7f597015d330 2023-12-13T23:56:00.575 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:00.575 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:56:00.576 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:56:00.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 msgr2=0x7f595406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 0x7f595406d680 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f596c007600 tx=0x7f596c00b040 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 msgr2=0x7f59700b71f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f596802b750 tx=0x7f59680046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 shutdown_connections 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f595406b1c0 0x7f595406d680 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f597015d330 0x7f59700b71f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59700d2d90 0x7f59701f6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 --2- 172.21.15.124:0/3309430943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59700d2910 0x7f59701569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.578+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 >> 172.21.15.124:0/3309430943 conn(0x7f5970072020 msgr2=0x7f59700b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.579+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 shutdown_connections 2023-12-13T23:56:00.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.579+0000 7f5977768640 1 -- 172.21.15.124:0/3309430943 wait complete. 2023-12-13T23:56:00.588 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:56:00.688 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.689+0000 7f81cb490640 1 Processor -- start 2023-12-13T23:56:00.688 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.689+0000 7f81cb490640 1 -- start start 2023-12-13T23:56:00.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81cb490640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f81c4143490 0x7f81c40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81cb490640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c415e2a0 con 0x7f81c415abf0 2023-12-13T23:56:00.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81cb490640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81c415b240 con 0x7f81c415e730 2023-12-13T23:56:00.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81cb490640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81c415b520 con 0x7f81c4143490 2023-12-13T23:56:00.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81c9a2d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f81c415abf0 0x7f81c40b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:38058/0 (socket says 172.21.15.124:38058) 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81c9a2d640 1 -- 172.21.15.124:0/976957414 learned_addr learned my addr 172.21.15.124:0/976957414 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81c922c640 1 --2- 172.21.15.124:0/976957414 >> v2:172.21.15.203:3300/0 conn(0x7f81c4143490 0x7f81c40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414203494 0 0) 0x7f81c415e2a0 con 0x7f81c415abf0 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0003520 con 0x7f81c415abf0 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113113020 0 0) 0x7f81b0003520 con 0x7f81c415abf0 2023-12-13T23:56:00.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 >> v2:172.21.15.203:3300/0 conn(0x7f81c4143490 msgr2=0x7f81c40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 --2- 172.21.15.124:0/976957414 >> v2:172.21.15.203:3300/0 conn(0x7f81c4143490 0x7f81c40b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 >> v1:172.21.15.203:6789/0 conn(0x7f81c415e730 legacy=0x7f81c415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c415dc50 con 0x7f81c415abf0 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f81bc0019f0 con 0x7f81c415abf0 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 >> v1:172.21.15.124:6789/0 conn(0x7f81c415abf0 legacy=0x7f81c40b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81b0003970 0x7f81b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b00047c0 0x7f81b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 0x7f81b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c415b520 con 0x7f81b0005180 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81b000c020 con 0x7f81b0003970 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.690+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81b000c1e0 con 0x7f81b00047c0 2023-12-13T23:56:00.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 0x7f81b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c922c640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81b0003970 0x7f81b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c8a2b640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b00047c0 0x7f81b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 -- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b00047c0 msgr2=0x7f81b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b00047c0 0x7f81b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 -- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81b0003970 msgr2=0x7f81b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81b0003970 0x7f81b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 -- 172.21.15.124:0/976957414 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b0003520 con 0x7f81b0005180 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c8a2b640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b00047c0 0x7f81b0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.691+0000 7f81c9a2d640 1 -- 172.21.15.124:0/976957414 reap_dead start 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81c9a2d640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 0x7f81b000b980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00a830 tx=0x7f81b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=990e3daa87ae7b53 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0072d0 con 0x7f81b0005180 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81bc02f070 con 0x7f81b0005180 2023-12-13T23:56:00.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81bbfff640 1 -- 172.21.15.124:0/976957414 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc0076b0 con 0x7f81b0005180 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 -- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 msgr2=0x7f81b000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 0x7f81b000b980 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f81bc00a830 tx=0x7f81b00057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 -- 172.21.15.124:0/976957414 shutdown_connections 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 --2- 172.21.15.124:0/976957414 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81b0005180 0x7f81b000b980 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 -- 172.21.15.124:0/976957414 >> 172.21.15.124:0/976957414 conn(0x7f81c406a4d0 msgr2=0x7f81c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 -- 172.21.15.124:0/976957414 shutdown_connections 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.692+0000 7f81cb490640 1 -- 172.21.15.124:0/976957414 wait complete. 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.693+0000 7f81cb490640 1 Processor -- start 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.693+0000 7f81cb490640 1 -- start start 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b0005180 0x7f81c41363c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81c415bba0 0x7f81c415c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c4138290 con 0x7f81c413ab70 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c4137c60 con 0x7f81c415bba0 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81cb490640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81c413a740 con 0x7f81b0005180 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81c8a2b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81c8a2b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48434/0 (socket says 172.21.15.124:48434) 2023-12-13T23:56:00.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81c8a2b640 1 -- 172.21.15.124:0/3909904460 learned_addr learned my addr 172.21.15.124:0/3909904460 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81c8a2b640 1 -- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b0005180 msgr2=0x7f81c41363c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.694+0000 7f81c8a2b640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b0005180 0x7f81c41363c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81c8a2b640 1 -- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81c415bba0 msgr2=0x7f81c415c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81c8a2b640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81c415bba0 0x7f81c415c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81c8a2b640 1 -- 172.21.15.124:0/3909904460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81c413d870 con 0x7f81c413ab70 2023-12-13T23:56:00.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81c8a2b640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f81b400e460 tx=0x7f81b400e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b40052e0 con 0x7f81c413ab70 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b4005440 con 0x7f81c413ab70 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c413db50 con 0x7f81c413ab70 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b4003830 con 0x7f81c413ab70 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.695+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c413aff0 con 0x7f81c413ab70 2023-12-13T23:56:00.709 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.696+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81c40cbed0 con 0x7f81c413ab70 2023-12-13T23:56:00.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.696+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f81b4003990 con 0x7f81c413ab70 2023-12-13T23:56:00.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.697+0000 7f81b9ffb640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 0x7f81a806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:00.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.697+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f81b40896d0 con 0x7f81c413ab70 2023-12-13T23:56:00.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.697+0000 7f81c922c640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 0x7f81a806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:00.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.697+0000 7f81c922c640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 0x7f81a806d6b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f81c41f2650 tx=0x7f81c0009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:00.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.701+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f81b4054410 con 0x7f81c413ab70 2023-12-13T23:56:00.902 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.904+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81c4069400 con 0x7f81c413ab70 2023-12-13T23:56:00.903 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.904+0000 7f81b9ffb640 1 -- 172.21.15.124:0/3909904460 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4916 (secure 0 0 0) 0x7f81b4057a30 con 0x7f81c413ab70 2023-12-13T23:56:00.903 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:00.903 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2947940333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2947940333},{"type":"v1","addr":"172.21.15.203:6836","nonce":2947940333}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4293,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6834/1311858518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":1311858518},{"type":"v1","addr":"172.21.15.124:6834","nonce":1311858518}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4296,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/141371148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6833","nonce":141371148},{"type":"v1","addr":"172.21.15.124:6835","nonce":141371148}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:55:12.125635+0000","modified":"2023-12-13T23:55:14.093059+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4308":{"gid":4308,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/1545879646","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":1545879646},{"type":"v1","addr":"172.21.15.203:6837","nonce":1545879646}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-12-13T23:56:00.903 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 6 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 msgr2=0x7f81a806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 0x7f81a806d6b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f81c41f2650 tx=0x7f81c0009320 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 msgr2=0x7f81c415b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f81b400e460 tx=0x7f81b400e920 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 shutdown_connections 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f81a806b1f0 0x7f81a806d6b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f81c415bba0 0x7f81c415c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f81c413ab70 0x7f81c415b660 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 --2- 172.21.15.124:0/3909904460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f81b0005180 0x7f81c41363c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:00.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 >> 172.21.15.124:0/3909904460 conn(0x7f81c406a4d0 msgr2=0x7f81c41429d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:00.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 shutdown_connections 2023-12-13T23:56:00.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:00.907+0000 7f81cb490640 1 -- 172.21.15.124:0/3909904460 wait complete. 2023-12-13T23:56:00.917 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:01.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.018+0000 7f5615017640 1 Processor -- start 2023-12-13T23:56:01.017 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.018+0000 7f5615017640 1 -- start start 2023-12-13T23:56:01.018 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f5615017640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5610158ba0 0x7f561015c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.018 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f5615017640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 0x7f56100ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f5615017640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56100adbb0 con 0x7f5610071c80 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f5615017640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56100b4520 con 0x7f5610158ba0 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f5615017640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561015c6a0 con 0x7f56100ac370 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f560ed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5610158ba0 0x7f561015c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f560f59e640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f5610071c80 0x7f56100b2200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:38070/0 (socket says 172.21.15.124:38070) 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.019+0000 7f560f59e640 1 -- 172.21.15.124:0/2680024656 learned_addr learned my addr 172.21.15.124:0/2680024656 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560ed9d640 1 -- 172.21.15.124:0/2680024656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 msgr2=0x7f56100ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560e59c640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 0x7f56100ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560dd9b640 1 -- 172.21.15.124:0/2680024656 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678878673 0 0) 0x7f56100adbb0 con 0x7f5610071c80 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560ed9d640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 0x7f56100ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560ed9d640 1 -- 172.21.15.124:0/2680024656 >> v1:172.21.15.124:6789/0 conn(0x7f5610071c80 legacy=0x7f56100b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560ed9d640 1 -- 172.21.15.124:0/2680024656 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56100b3f60 con 0x7f5610158ba0 2023-12-13T23:56:01.019 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560e59c640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 0x7f56100ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:01.020 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.020+0000 7f560ed9d640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f561015c160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f56080097a0 tx=0x7f560802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858b722375b97d4a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f560dd9b640 1 -- 172.21.15.124:0/2680024656 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560802e070 con 0x7f5610158ba0 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f560dd9b640 1 -- 172.21.15.124:0/2680024656 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56080315a0 con 0x7f5610158ba0 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f560dd9b640 1 -- 172.21.15.124:0/2680024656 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5608031930 con 0x7f5610158ba0 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 -- 172.21.15.124:0/2680024656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 msgr2=0x7f561015c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f561015c160 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f56080097a0 tx=0x7f560802b840 comp rx=0 tx=0).stop 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 -- 172.21.15.124:0/2680024656 shutdown_connections 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f56100ac370 0x7f56100ae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 --2- 172.21.15.124:0/2680024656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f561015c160 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 -- 172.21.15.124:0/2680024656 >> 172.21.15.124:0/2680024656 conn(0x7f56100b4720 msgr2=0x7f56100b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 -- 172.21.15.124:0/2680024656 shutdown_connections 2023-12-13T23:56:01.021 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.021+0000 7f5615017640 1 -- 172.21.15.124:0/2680024656 wait complete. 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 Processor -- start 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 -- start start 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5610071c80 0x7f561009cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56100ac370 0x7f5610099d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f561009d280 con 0x7f5610158ba0 2023-12-13T23:56:01.023 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610140570 con 0x7f56100ac370 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f5615017640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610140850 con 0x7f5610071c80 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f560f59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f560f59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48456/0 (socket says 172.21.15.124:48456) 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.023+0000 7f560f59e640 1 -- 172.21.15.124:0/2906475797 learned_addr learned my addr 172.21.15.124:0/2906475797 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560ed9d640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5610071c80 0x7f561009cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560e59c640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56100ac370 0x7f5610099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 -- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5610071c80 msgr2=0x7f561009cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5610071c80 0x7f561009cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 -- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56100ac370 msgr2=0x7f5610099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56100ac370 0x7f5610099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 -- 172.21.15.124:0/2906475797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5608009400 con 0x7f5610158ba0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f560f59e640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f560000c410 tx=0x7f560000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56000074e0 con 0x7f5610158ba0 2023-12-13T23:56:01.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600007640 con 0x7f5610158ba0 2023-12-13T23:56:01.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610140b30 con 0x7f5610158ba0 2023-12-13T23:56:01.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600004a20 con 0x7f5610158ba0 2023-12-13T23:56:01.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.024+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56101410f0 con 0x7f5610158ba0 2023-12-13T23:56:01.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.025+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5610069400 con 0x7f5610158ba0 2023-12-13T23:56:01.025 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.025+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5600004b80 con 0x7f5610158ba0 2023-12-13T23:56:01.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.026+0000 7f55f77fe640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 0x7f55fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.026+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5600089910 con 0x7f5610158ba0 2023-12-13T23:56:01.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.026+0000 7f560ed9d640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 0x7f55fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.026+0000 7f560ed9d640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 0x7f55fc06d680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5608036000 tx=0x7f5608008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.030+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5600054530 con 0x7f5610158ba0 2023-12-13T23:56:01.197 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.198+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5610070b70 con 0x7f5610158ba0 2023-12-13T23:56:01.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.199+0000 7f55f77fe640 1 -- 172.21.15.124:0/2906475797 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16815 (secure 0 0 0) 0x7f5600001060 con 0x7f5610158ba0 2023-12-13T23:56:01.198 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:01.198 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":23,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:55:53.877550+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:55:10.129448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-12-13T23:55:11.064925+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":3,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.16","mappings":[{"from":5,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:01.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 msgr2=0x7f55fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 0x7f55fc06d680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5608036000 tx=0x7f5608008030 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 msgr2=0x7f56100b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f560000c410 tx=0x7f560000c8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 shutdown_connections 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f55fc06b1c0 0x7f55fc06d680 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f56100ac370 0x7f5610099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5610158ba0 0x7f56100b0ae0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 --2- 172.21.15.124:0/2906475797 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5610071c80 0x7f561009cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 >> 172.21.15.124:0/2906475797 conn(0x7f56100b4720 msgr2=0x7f561015e340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 shutdown_connections 2023-12-13T23:56:01.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.202+0000 7f5615017640 1 -- 172.21.15.124:0/2906475797 wait complete. 2023-12-13T23:56:01.212 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-13T23:56:01.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 Processor -- start 2023-12-13T23:56:01.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 -- start start 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71c4143490 0x7f71c40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 0x7f71c415a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c415b520 con 0x7f71c4143490 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71c4143870 con 0x7f71c415a4e0 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.314+0000 7f71caebb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c415e2a0 con 0x7f71c415e730 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71c4143490 0x7f71c40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f71c4143490 0x7f71c40b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48460/0 (socket says 172.21.15.124:48460) 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 -- 172.21.15.124:0/979645324 learned_addr learned my addr 172.21.15.124:0/979645324 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c9458640 1 --2- 172.21.15.124:0/979645324 >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 0x7f71c415a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c3fff640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 -- 172.21.15.124:0/979645324 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 msgr2=0x7f71c40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 -- 172.21.15.124:0/979645324 >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 msgr2=0x7f71c415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 --2- 172.21.15.124:0/979645324 >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 0x7f71c415a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c8c57640 1 -- 172.21.15.124:0/979645324 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c415b240 con 0x7f71c4143490 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.315+0000 7f71c9458640 1 --2- 172.21.15.124:0/979645324 >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 0x7f71c415a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71c8c57640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c4143490 0x7f71c40b0cf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f71c415eb90 tx=0x7f71bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33a05ab2fac488ff server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71c37fe640 1 -- 172.21.15.124:0/979645324 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc02ca70 con 0x7f71c4143490 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71c37fe640 1 -- 172.21.15.124:0/979645324 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71bc02cbd0 con 0x7f71c4143490 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71c37fe640 1 -- 172.21.15.124:0/979645324 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71bc0343f0 con 0x7f71c4143490 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 -- 172.21.15.124:0/979645324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c4143490 msgr2=0x7f71c40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c4143490 0x7f71c40b0cf0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f71c415eb90 tx=0x7f71bc02bed0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 -- 172.21.15.124:0/979645324 shutdown_connections 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 --2- 172.21.15.124:0/979645324 >> v2:172.21.15.203:3300/0 conn(0x7f71c415a4e0 0x7f71c415a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 --2- 172.21.15.124:0/979645324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c4143490 0x7f71c40b0cf0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 -- 172.21.15.124:0/979645324 >> 172.21.15.124:0/979645324 conn(0x7f71c406a4d0 msgr2=0x7f71c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 -- 172.21.15.124:0/979645324 shutdown_connections 2023-12-13T23:56:01.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.316+0000 7f71caebb640 1 -- 172.21.15.124:0/979645324 wait complete. 2023-12-13T23:56:01.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.317+0000 7f71caebb640 1 Processor -- start 2023-12-13T23:56:01.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.317+0000 7f71caebb640 1 -- start start 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c4143490 0x7f71c413ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c41e4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c4139df0 con 0x7f71c415a4e0 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71c413a0d0 con 0x7f71c4143490 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71caebb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71c413a3b0 con 0x7f71c415e730 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71c3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71c3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48476/0 (socket says 172.21.15.124:48476) 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.318+0000 7f71c3fff640 1 -- 172.21.15.124:0/2885783996 learned_addr learned my addr 172.21.15.124:0/2885783996 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c8c57640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c4143490 0x7f71c413ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c9458640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c41e4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 -- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 msgr2=0x7f71c41e4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c41e4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 -- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c4143490 msgr2=0x7f71c413ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c4143490 0x7f71c413ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 -- 172.21.15.124:0/2885783996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71bc009b30 con 0x7f71c415a4e0 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c3fff640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00d5f0 tx=0x7f71ac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac0125f0 con 0x7f71c415a4e0 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71ac012750 con 0x7f71c415a4e0 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71c413a690 con 0x7f71c415a4e0 2023-12-13T23:56:01.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac012b00 con 0x7f71c415a4e0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.319+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71c41e4c70 con 0x7f71c415a4e0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.320+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71c4069400 con 0x7f71c415a4e0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.320+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f71ac012c60 con 0x7f71c415a4e0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.321+0000 7f71c17fa640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 0x7f71a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.321+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f71ac089e90 con 0x7f71c415a4e0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.321+0000 7f71c8c57640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 0x7f71a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.321+0000 7f71c8c57640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 0x7f71a406d680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f71bc035000 tx=0x7f71bc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.325+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f71ac054bd0 con 0x7f71c415a4e0 2023-12-13T23:56:01.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.524+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f71c40b2820 con 0x7f71c415a4e0 2023-12-13T23:56:01.523 INFO:tasks.ceph.mon.a.smithi124.stderr:2023-12-13T23:56:01.525+0000 7ffbf0477640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-13T23:56:01.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.526+0000 7f71c17fa640 1 -- 172.21.15.124:0/2885783996 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) v1 ==== 161+0+0 (secure 0 0 0) 0x7f71ac0581f0 con 0x7f71c415a4e0 2023-12-13T23:56:01.524 INFO:teuthology.orchestra.run.smithi124.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 msgr2=0x7f71a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 0x7f71a406d680 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f71bc035000 tx=0x7f71bc030000 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 msgr2=0x7f71c413b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f71ac00d5f0 tx=0x7f71ac00dab0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 shutdown_connections 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71a406b1c0 0x7f71a406d680 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71c415e730 0x7f71c41e4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71c415a4e0 0x7f71c413b3d0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 --2- 172.21.15.124:0/2885783996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71c4143490 0x7f71c413ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 >> 172.21.15.124:0/2885783996 conn(0x7f71c406a4d0 msgr2=0x7f71c4070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 shutdown_connections 2023-12-13T23:56:01.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.528+0000 7f71caebb640 1 -- 172.21.15.124:0/2885783996 wait complete. 2023-12-13T23:56:01.530 INFO:tasks.ceph.mds.d.smithi203.stderr: -2829> 2023-12-13T23:55:17.763+0000 7f515f001640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:56:01.538 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-13T23:56:01.561 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2023-12-13T23:56:01.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 Processor -- start 2023-12-13T23:56:01.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 -- start start 2023-12-13T23:56:01.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb35015d680 0x7fb3500c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb3500cb520 0x7fb3500cc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35015da60 con 0x7fb3500cd850 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3500b1d90 con 0x7fb35015d680 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb355a61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3500ccbb0 con 0x7fb3500cb520 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.641+0000 7fb34effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb35015d680 0x7fb3500c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb35015d680 0x7fb3500c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48492/0 (socket says 172.21.15.124:48492) 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 -- 172.21.15.124:0/4072348850 learned_addr learned my addr 172.21.15.124:0/4072348850 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34e7fc640 1 --2- 172.21.15.124:0/4072348850 >> v2:172.21.15.203:3300/0 conn(0x7fb3500cb520 0x7fb3500cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 -- 172.21.15.124:0/4072348850 >> v2:172.21.15.203:3300/0 conn(0x7fb3500cb520 msgr2=0x7fb3500cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 --2- 172.21.15.124:0/4072348850 >> v2:172.21.15.203:3300/0 conn(0x7fb3500cb520 0x7fb3500cc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 -- 172.21.15.124:0/4072348850 >> v1:172.21.15.203:6789/0 conn(0x7fb3500cd850 legacy=0x7fb3500cdc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:01.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 -- 172.21.15.124:0/4072348850 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3500cb990 con 0x7fb35015d680 2023-12-13T23:56:01.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.642+0000 7fb34effd640 1 --2- 172.21.15.124:0/4072348850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c9e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb3500707b0 tx=0x7fb34002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e7df316440ab49 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb34dffb640 1 -- 172.21.15.124:0/4072348850 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34002ea00 con 0x7fb35015d680 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb34dffb640 1 -- 172.21.15.124:0/4072348850 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34002eb60 con 0x7fb35015d680 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb34dffb640 1 -- 172.21.15.124:0/4072348850 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340009600 con 0x7fb35015d680 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 -- 172.21.15.124:0/4072348850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 msgr2=0x7fb3500c9e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 --2- 172.21.15.124:0/4072348850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c9e80 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fb3500707b0 tx=0x7fb34002e060 comp rx=0 tx=0).stop 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 -- 172.21.15.124:0/4072348850 shutdown_connections 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 --2- 172.21.15.124:0/4072348850 >> v2:172.21.15.203:3300/0 conn(0x7fb3500cb520 0x7fb3500cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 --2- 172.21.15.124:0/4072348850 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c9e80 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 -- 172.21.15.124:0/4072348850 >> 172.21.15.124:0/4072348850 conn(0x7fb3500b0f00 msgr2=0x7fb3500b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 -- 172.21.15.124:0/4072348850 shutdown_connections 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.643+0000 7fb355a61640 1 -- 172.21.15.124:0/4072348850 wait complete. 2023-12-13T23:56:01.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.644+0000 7fb355a61640 1 Processor -- start 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 -- start start 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3500cb520 0x7fb35014e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 0x7fb3501f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb35014ed20 con 0x7fb35015d680 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501f3f50 con 0x7fb3500cd850 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb355a61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3501f3580 con 0x7fb3500cb520 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb34f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb34f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48496/0 (socket says 172.21.15.124:48496) 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.645+0000 7fb34f7fe640 1 -- 172.21.15.124:0/3447969224 learned_addr learned my addr 172.21.15.124:0/3447969224 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34e7fc640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 0x7fb3501f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34effd640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3500cb520 0x7fb35014e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 -- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3500cb520 msgr2=0x7fb35014e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3500cb520 0x7fb35014e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 -- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 msgr2=0x7fb3501f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 0x7fb3501f3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 -- 172.21.15.124:0/3447969224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34000bd40 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34e7fc640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 0x7fb3501f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb34f7fe640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb34800b410 tx=0x7fb34800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348004020 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb348004180 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3500c8fd0 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb348011630 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.646+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3501f3210 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.647+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3501f3730 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.647+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb348011790 con 0x7fb35015d680 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.648+0000 7fb337fff640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 0x7fb33c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:01.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.648+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fb348089710 con 0x7fb35015d680 2023-12-13T23:56:01.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.648+0000 7fb34effd640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 0x7fb33c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:01.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.649+0000 7fb34effd640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 0x7fb33c06d680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb340037000 tx=0x7fb340032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:01.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.652+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb3501f3730 con 0x7fb35015d680 2023-12-13T23:56:01.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:01.852+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb3501f36f0 con 0x7fb35015d680 2023-12-13T23:56:02.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.527+0000 7fb337fff640 1 -- 172.21.15.124:0/3447969224 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb348054380 con 0x7fb35015d680 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 msgr2=0x7fb33c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 0x7fb33c06d680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fb340037000 tx=0x7fb340032000 comp rx=0 tx=0).stop 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 msgr2=0x7fb3500c8580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fb34800b410 tx=0x7fb34800b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 shutdown_connections 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb33c06b1c0 0x7fb33c06d680 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb3500cd850 0x7fb3501f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb35015d680 0x7fb3500c8580 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 --2- 172.21.15.124:0/3447969224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb3500cb520 0x7fb35014e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 >> 172.21.15.124:0/3447969224 conn(0x7fb3500b0f00 msgr2=0x7fb35006de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:02.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 shutdown_connections 2023-12-13T23:56:02.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.529+0000 7fb355a61640 1 -- 172.21.15.124:0/3447969224 wait complete. 2023-12-13T23:56:02.540 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-13T23:56:02.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.647+0000 7fbde4f89640 1 Processor -- start 2023-12-13T23:56:02.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.647+0000 7fbde4f89640 1 -- start start 2023-12-13T23:56:02.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbde4f89640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbde4f89640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015abf0 0x7fbde00b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbde4f89640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde015e2a0 con 0x7fbde015e730 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbde4f89640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde015b520 con 0x7fbde015abf0 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbde4f89640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0143870 con 0x7fbde0143490 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015abf0 0x7fbde00b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015abf0 0x7fbde00b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48508/0 (socket says 172.21.15.124:48508) 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 -- 172.21.15.124:0/3042376098 learned_addr learned my addr 172.21.15.124:0/3042376098 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdde59c640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 -- 172.21.15.124:0/3042376098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 msgr2=0x7fbde00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 -- 172.21.15.124:0/3042376098 >> v1:172.21.15.203:6789/0 conn(0x7fbde015e730 legacy=0x7fbde015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.648+0000 7fbdded9d640 1 -- 172.21.15.124:0/3042376098 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde015b240 con 0x7fbde015abf0 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbdded9d640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde00b1230 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009e50 tx=0x7fbdd802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3daf3ff88290c08 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:02.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbdddd9b640 1 -- 172.21.15.124:0/3042376098 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd802cb00 con 0x7fbde015abf0 2023-12-13T23:56:02.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbdddd9b640 1 -- 172.21.15.124:0/3042376098 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd802cc60 con 0x7fbde015abf0 2023-12-13T23:56:02.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbdddd9b640 1 -- 172.21.15.124:0/3042376098 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80343f0 con 0x7fbde015abf0 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbde4f89640 1 -- 172.21.15.124:0/3042376098 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 msgr2=0x7fbde00b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.649+0000 7fbde4f89640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde00b1230 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8009e50 tx=0x7fbdd802bf60 comp rx=0 tx=0).stop 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 -- 172.21.15.124:0/3042376098 shutdown_connections 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde00b1230 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 --2- 172.21.15.124:0/3042376098 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 -- 172.21.15.124:0/3042376098 >> 172.21.15.124:0/3042376098 conn(0x7fbde006a4d0 msgr2=0x7fbde006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 -- 172.21.15.124:0/3042376098 shutdown_connections 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.650+0000 7fbde4f89640 1 -- 172.21.15.124:0/3042376098 wait complete. 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.651+0000 7fbde4f89640 1 Processor -- start 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.651+0000 7fbde4f89640 1 -- start start 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015e730 0x7fbde00708d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b16d0 con 0x7fbde015abf0 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b0b50 con 0x7fbde015e730 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde00b0e30 con 0x7fbde0143490 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48512/0 (socket says 172.21.15.124:48512) 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 -- 172.21.15.124:0/807452170 learned_addr learned my addr 172.21.15.124:0/807452170 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdded9d640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015e730 0x7fbde00708d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdde59c640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 -- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 msgr2=0x7fbde013e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 -- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015e730 msgr2=0x7fbde00708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015e730 0x7fbde00708d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 -- 172.21.15.124:0/807452170 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009b30 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd7fff640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c400 tx=0x7fbdc800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc80052d0 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8005430 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc8003830 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde00b1110 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.652+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde013edb0 con 0x7fbde015abf0 2023-12-13T23:56:02.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.653+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde0069400 con 0x7fbde015abf0 2023-12-13T23:56:02.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.654+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbdc8003990 con 0x7fbde015abf0 2023-12-13T23:56:02.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.654+0000 7fbdd6ffd640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 0x7fbdc006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:02.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.654+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5911+0+0 (secure 0 0 0) 0x7fbdc8088fc0 con 0x7fbde015abf0 2023-12-13T23:56:02.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.655+0000 7fbdde59c640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 0x7fbdc006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:02.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.655+0000 7fbdde59c640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 0x7fbdc006d680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0006fd0 tx=0x7fbdd0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:02.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.658+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbdc80bbe90 con 0x7fbde015abf0 2023-12-13T23:56:02.824 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:02.826+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbde00b2820 con 0x7fbde015abf0 2023-12-13T23:56:03.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.742+0000 7fbdd6ffd640 1 -- 172.21.15.124:0/807452170 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbdc80170d0 con 0x7fbde015abf0 2023-12-13T23:56:03.741 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' removed 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 msgr2=0x7fbdc006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 0x7fbdc006d680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0006fd0 tx=0x7fbdd0008040 comp rx=0 tx=0).stop 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 msgr2=0x7fbde0070390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fbdc800c400 tx=0x7fbdc800c8c0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 shutdown_connections 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1c0 0x7fbdc006d680 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015e730 0x7fbde00708d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015abf0 0x7fbde0070390 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 --2- 172.21.15.124:0/807452170 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 >> 172.21.15.124:0/807452170 conn(0x7fbde006a4d0 msgr2=0x7fbde006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:03.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 shutdown_connections 2023-12-13T23:56:03.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.745+0000 7fbde4f89640 1 -- 172.21.15.124:0/807452170 wait complete. 2023-12-13T23:56:03.758 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-13T23:56:03.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 Processor -- start 2023-12-13T23:56:03.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 -- start start 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8f1c0d2910 0x7f8f1c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0b2b10 con 0x7f8f1c15d330 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1c0d1590 con 0x7f8f1c0d3730 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.871+0000 7f8f22bb6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1c0d3bc0 con 0x7f8f1c0d2910 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8f1c15d330 0x7f8f1c0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:38084/0 (socket says 172.21.15.124:38084) 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 -- 172.21.15.124:0/1240854037 learned_addr learned my addr 172.21.15.124:0/1240854037 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513538984 0 0) 0x7f8f1c0b2b10 con 0x7f8f1c15d330 2023-12-13T23:56:03.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f0c003520 con 0x7f8f1c15d330 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1bfff640 1 --2- 172.21.15.124:0/1240854037 >> v2:172.21.15.203:3300/0 conn(0x7f8f1c0d2910 0x7f8f1c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 813639225 0 0) 0x7f8f0c003520 con 0x7f8f1c15d330 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 >> v2:172.21.15.203:3300/0 conn(0x7f8f1c0d2910 msgr2=0x7f8f1c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 --2- 172.21.15.124:0/1240854037 >> v2:172.21.15.203:3300/0 conn(0x7f8f1c0d2910 0x7f8f1c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 >> v1:172.21.15.203:6789/0 conn(0x7f8f1c0d3730 legacy=0x7f8f1c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c0d1870 con 0x7f8f1c15d330 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8f10001670 con 0x7f8f1c15d330 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 >> v1:172.21.15.124:6789/0 conn(0x7f8f1c15d330 legacy=0x7f8f1c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 0x7f8f0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c0047c0 0x7f8f0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f0c005180 0x7f8f0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c0d3bc0 con 0x7f8f0c003970 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c00c120 con 0x7f8f0c0047c0 2023-12-13T23:56:03.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0c00c2e0 con 0x7f8f0c005180 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 0x7f8f0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f1bfff640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c0047c0 0x7f8f0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f21153640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f0c005180 0x7f8f0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 -- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f0c005180 msgr2=0x7f8f0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f0c005180 0x7f8f0c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 -- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c0047c0 msgr2=0x7f8f0c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c0047c0 0x7f8f0c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f20952640 1 -- 172.21.15.124:0/1240854037 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c003520 con 0x7f8f0c003970 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.872+0000 7f8f21153640 1 -- 172.21.15.124:0/1240854037 reap_dead start 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.873+0000 7f8f20952640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 0x7f8f0c003d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c240 tx=0x7f8f0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2705b9bab5b2cabc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.873+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f10005ad0 con 0x7f8f0c003970 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.873+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f10005c30 con 0x7f8f0c003970 2023-12-13T23:56:03.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.873+0000 7f8f1b7fe640 1 -- 172.21.15.124:0/1240854037 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1002caf0 con 0x7f8f0c003970 2023-12-13T23:56:03.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 -- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 msgr2=0x7f8f0c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 0x7f8f0c003d90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8f1000c240 tx=0x7f8f0c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 -- 172.21.15.124:0/1240854037 shutdown_connections 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/1240854037 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f0c003970 0x7f8f0c003d90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 -- 172.21.15.124:0/1240854037 >> 172.21.15.124:0/1240854037 conn(0x7f8f1c072020 msgr2=0x7f8f1c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 -- 172.21.15.124:0/1240854037 shutdown_connections 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.874+0000 7f8f22bb6640 1 -- 172.21.15.124:0/1240854037 wait complete. 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.875+0000 7f8f22bb6640 1 Processor -- start 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.875+0000 7f8f22bb6640 1 -- start start 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c003970 0x7f8f1c152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f1c0c98b0 0x7f8f1c15f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c154530 con 0x7f8f1c1533d0 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c159530 con 0x7f8f0c003970 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f22bb6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1c153d20 con 0x7f8f1c0c98b0 2023-12-13T23:56:03.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48534/0 (socket says 172.21.15.124:48534) 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 -- 172.21.15.124:0/3964079092 learned_addr learned my addr 172.21.15.124:0/3964079092 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 -- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f1c0c98b0 msgr2=0x7f8f1c15f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f1c0c98b0 0x7f8f1c15f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 -- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c003970 msgr2=0x7f8f1c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c003970 0x7f8f1c152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.876+0000 7f8f1bfff640 1 -- 172.21.15.124:0/3964079092 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1c1597d0 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f1bfff640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8f08011700 tx=0x7f8f08011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0800f9c0 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f0800fb20 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0800fe90 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f1c153f70 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.877+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f1c159c60 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.878+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f1c069400 con 0x7f8f1c1533d0 2023-12-13T23:56:03.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.878+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8f08015460 con 0x7f8f1c1533d0 2023-12-13T23:56:03.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.879+0000 7f8f197fa640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 0x7f8f0006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:03.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.879+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f8f08089e50 con 0x7f8f1c1533d0 2023-12-13T23:56:03.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.879+0000 7f8f20952640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 0x7f8f0006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:03.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.879+0000 7f8f20952640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 0x7f8f0006d680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8f10034000 tx=0x7f8f10039040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:03.885 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:03.887+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8f08054be0 con 0x7f8f1c1533d0 2023-12-13T23:56:04.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.063+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8f1c159960 con 0x7f8f1c1533d0 2023-12-13T23:56:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.925+0000 7f8f197fa640 1 -- 172.21.15.124:0/3964079092 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8f08058200 con 0x7f8f1c1533d0 2023-12-13T23:56:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' removed 2023-12-13T23:56:04.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 msgr2=0x7f8f0006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:04.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 0x7f8f0006d680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8f10034000 tx=0x7f8f10039040 comp rx=0 tx=0).stop 2023-12-13T23:56:04.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 msgr2=0x7f8f1c15ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8f08011700 tx=0x7f8f08011bc0 comp rx=0 tx=0).stop 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 shutdown_connections 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8f0006b1c0 0x7f8f0006d680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8f1c0c98b0 0x7f8f1c15f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f1c1533d0 0x7f8f1c15ed00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 --2- 172.21.15.124:0/3964079092 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8f0c003970 0x7f8f1c152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 >> 172.21.15.124:0/3964079092 conn(0x7f8f1c072020 msgr2=0x7f8f1c0b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 shutdown_connections 2023-12-13T23:56:04.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:04.927+0000 7f8f22bb6640 1 -- 172.21.15.124:0/3964079092 wait complete. 2023-12-13T23:56:04.937 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-13T23:56:04.938 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-13T23:56:04.938 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-13T23:56:04.938 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:04.938 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-13T23:56:04.938 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-13T23:56:04.938 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-13T23:56:04.938 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:04.939 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-13T23:56:04.939 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-13T23:56:04.939 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-13T23:56:04.939 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:04.940 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-13T23:56:04.940 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-13T23:56:04.940 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-13T23:56:04.940 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:04.948 INFO:tasks.ceph.mds.c:Stopped 2023-12-13T23:56:04.948 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-13T23:56:04.952 INFO:tasks.ceph.mds.b:Stopped 2023-12-13T23:56:04.952 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-13T23:56:04.953 INFO:tasks.ceph.mds.d:Stopped 2023-12-13T23:56:04.953 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-13T23:56:04.992 INFO:tasks.ceph.mds.c:Started 2023-12-13T23:56:04.992 DEBUG:teuthology.parallel:result is None 2023-12-13T23:56:04.995 INFO:tasks.ceph.mds.b:Started 2023-12-13T23:56:04.995 DEBUG:teuthology.parallel:result is None 2023-12-13T23:56:04.996 INFO:tasks.ceph.mds.d:Started 2023-12-13T23:56:04.996 DEBUG:teuthology.parallel:result is None 2023-12-13T23:56:05.070 INFO:tasks.ceph.mds.c.smithi124.stdout:starting mds.c at 2023-12-13T23:56:05.079 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2023-12-13T23:56:05.079 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2023-12-13T23:56:05.080 INFO:tasks.ceph.mds.d.smithi203.stderr:2023-12-13T23:56:05.081+0000 7feef9fd4640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-12-13T23:56:08.984 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2023-12-13T23:56:10.948 INFO:tasks.ceph.mds.a:Stopped 2023-12-13T23:56:10.948 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-13T23:56:10.950 INFO:tasks.ceph.mds.a:Started 2023-12-13T23:56:10.951 DEBUG:teuthology.parallel:result is None 2023-12-13T23:56:10.951 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-13T23:56:11.039 INFO:tasks.ceph.mds.a.smithi124.stdout:starting mds.a at 2023-12-13T23:56:11.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.069+0000 7f0e83716640 1 Processor -- start 2023-12-13T23:56:11.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 -- start start 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c15d330 0x7f0e7c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7c0b2b10 con 0x7f0e7c0d2910 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e7c0d1590 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e83716640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0d1870 con 0x7f0e7c15d330 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e81cb3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e81cb3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42904/0 (socket says 172.21.15.124:42904) 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.070+0000 7f0e81cb3640 1 -- 172.21.15.124:0/4197177384 learned_addr learned my addr 172.21.15.124:0/4197177384 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e814b2640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c15d330 0x7f0e7c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e81cb3640 1 -- 172.21.15.124:0/4197177384 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c15d330 msgr2=0x7f0e7c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:11.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e81cb3640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c15d330 0x7f0e7c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:11.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e81cb3640 1 -- 172.21.15.124:0/4197177384 >> v1:172.21.15.203:6789/0 conn(0x7f0e7c0d2910 legacy=0x7f0e7c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:11.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e81cb3640 1 -- 172.21.15.124:0/4197177384 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e7c0d1b50 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.071+0000 7f0e81cb3640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009e50 tx=0x7f0e7402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8556d0dc77fb898e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e73fff640 1 -- 172.21.15.124:0/4197177384 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7402c9e0 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e73fff640 1 -- 172.21.15.124:0/4197177384 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7402cb40 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e73fff640 1 -- 172.21.15.124:0/4197177384 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e740343f0 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 -- 172.21.15.124:0/4197177384 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c0d2d90 msgr2=0x7f0e7c0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0e74009e50 tx=0x7f0e7402be40 comp rx=0 tx=0).stop 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 -- 172.21.15.124:0/4197177384 shutdown_connections 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0b6390 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:11.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 --2- 172.21.15.124:0/4197177384 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c15d330 0x7f0e7c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:11.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 -- 172.21.15.124:0/4197177384 >> 172.21.15.124:0/4197177384 conn(0x7f0e7c072020 msgr2=0x7f0e7c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:11.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 -- 172.21.15.124:0/4197177384 shutdown_connections 2023-12-13T23:56:11.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.072+0000 7f0e83716640 1 -- 172.21.15.124:0/4197177384 wait complete. 2023-12-13T23:56:11.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.073+0000 7f0e83716640 1 Processor -- start 2023-12-13T23:56:11.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 -- start start 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 0x7f0e7c1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0c7490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c0c7b50 con 0x7f0e7c15d330 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c15e4c0 con 0x7f0e7c0d2d90 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e83716640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7c15e710 con 0x7f0e7c0d2910 2023-12-13T23:56:11.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e81cb3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e81cb3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42920/0 (socket says 172.21.15.124:42920) 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e81cb3640 1 -- 172.21.15.124:0/2701082263 learned_addr learned my addr 172.21.15.124:0/2701082263 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.074+0000 7f0e80cb1640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0c7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e814b2640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 0x7f0e7c1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 -- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 msgr2=0x7f0e7c1f3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 0x7f0e7c1f3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 -- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e7c0d2d90 msgr2=0x7f0e7c0c7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0c7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 -- 172.21.15.124:0/2701082263 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e74009b30 con 0x7f0e7c15d330 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e814b2640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 0x7f0e7c1f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e81cb3640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0e74036040 tx=0x7f0e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7403e450 con 0x7f0e7c15d330 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e7403e5d0 con 0x7f0e7c15d330 2023-12-13T23:56:11.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7c15e9f0 con 0x7f0e7c15d330 2023-12-13T23:56:11.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7403e980 con 0x7f0e7c15d330 2023-12-13T23:56:11.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.075+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7c15efb0 con 0x7f0e7c15d330 2023-12-13T23:56:11.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.076+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c15f1d0 con 0x7f0e7c15d330 2023-12-13T23:56:11.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.077+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0e7403eae0 con 0x7f0e7c15d330 2023-12-13T23:56:11.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.077+0000 7f0e71ffb640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 0x7f0e6006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:11.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.077+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0e740ac4f0 con 0x7f0e7c15d330 2023-12-13T23:56:11.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.077+0000 7f0e814b2640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 0x7f0e6006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:11.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.078+0000 7f0e814b2640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 0x7f0e6006d680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0e78006fd0 tx=0x7f0e78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:11.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.081+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0e74077520 con 0x7f0e7c15d330 2023-12-13T23:56:11.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:11.260+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0e7c069400 con 0x7f0e7c15d330 2023-12-13T23:56:11.818 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:56:11.821 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:56:12.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.950+0000 7f0e71ffb640 1 -- 172.21.15.124:0/2701082263 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v31) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0e7407ab40 con 0x7f0e7c15d330 2023-12-13T23:56:12.949 INFO:teuthology.orchestra.run.smithi124.stderr: removed all blocklist entries 2023-12-13T23:56:12.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 msgr2=0x7f0e6006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:12.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 0x7f0e6006d680 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0e78006fd0 tx=0x7f0e78008040 comp rx=0 tx=0).stop 2023-12-13T23:56:12.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 msgr2=0x7f0e7c1604a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:12.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0e74036040 tx=0x7f0e74004a40 comp rx=0 tx=0).stop 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 shutdown_connections 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0e6006b1c0 0x7f0e6006d680 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0e7c0d2d90 0x7f0e7c0c7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0e7c15d330 0x7f0e7c1604a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 --2- 172.21.15.124:0/2701082263 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0e7c0d2910 0x7f0e7c1f3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 >> 172.21.15.124:0/2701082263 conn(0x7f0e7c072020 msgr2=0x7f0e7c0cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 shutdown_connections 2023-12-13T23:56:12.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:12.953+0000 7f0e83716640 1 -- 172.21.15.124:0/2701082263 wait complete. 2023-12-13T23:56:12.965 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-13T23:56:13.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.070+0000 7f0a3f9ce640 1 Processor -- start 2023-12-13T23:56:13.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.070+0000 7f0a3f9ce640 1 -- start start 2023-12-13T23:56:13.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3f9ce640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0a38143490 0x7f0a380b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3f9ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a380b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3f9ce640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3815e2a0 con 0x7f0a3815a4e0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3f9ce640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3815b520 con 0x7f0a38143490 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3f9ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38143870 con 0x7f0a3815e730 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3df6b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0a3815a4e0 0x7f0a3815a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57818/0 (socket says 172.21.15.124:57818) 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3df6b640 1 -- 172.21.15.124:0/3459746896 learned_addr learned my addr 172.21.15.124:0/3459746896 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3d76a640 1 --2- 172.21.15.124:0/3459746896 >> v2:172.21.15.203:3300/0 conn(0x7f0a38143490 0x7f0a380b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852693762 0 0) 0x7f0a3815e2a0 con 0x7f0a3815a4e0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a380b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a24003520 con 0x7f0a3815a4e0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3449344891 0 0) 0x7f0a24003520 con 0x7f0a3815a4e0 2023-12-13T23:56:13.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 msgr2=0x7f0a380b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a380b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 >> v2:172.21.15.203:3300/0 conn(0x7f0a38143490 msgr2=0x7f0a380b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 --2- 172.21.15.124:0/3459746896 >> v2:172.21.15.203:3300/0 conn(0x7f0a38143490 0x7f0a380b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3815b240 con 0x7f0a3815a4e0 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.071+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0a300019f0 con 0x7f0a3815a4e0 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 >> v1:172.21.15.124:6789/0 conn(0x7f0a3815a4e0 legacy=0x7f0a3815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a24003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a24003ef0 0x7f0a240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a24004d00 0x7f0a24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3815b520 con 0x7f0a24003ef0 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38143870 con 0x7f0a24004d00 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2400b980 con 0x7f0a3815e730 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a24003ef0 0x7f0a240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3df6b640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a24004d00 0x7f0a24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3d76a640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a24003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 -- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 msgr2=0x7f0a24003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a3815e730 0x7f0a24003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 -- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a24004d00 msgr2=0x7f0a24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a24004d00 0x7f0a24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.072+0000 7f0a3cf69640 1 -- 172.21.15.124:0/3459746896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a24003520 con 0x7f0a24003ef0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a24003ef0 0x7f0a240047c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009ac0 tx=0x7f0a240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b0f2eee7789817 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2802c6b0 con 0x7f0a24003ef0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2802c810 con 0x7f0a24003ef0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a2ffff640 1 -- 172.21.15.124:0/3459746896 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2802cba0 con 0x7f0a24003ef0 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/3459746896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a24003ef0 msgr2=0x7f0a240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/3459746896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a24003ef0 0x7f0a240047c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0a28009ac0 tx=0x7f0a240057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3df6b640 1 -- 172.21.15.124:0/3459746896 reap_dead start 2023-12-13T23:56:13.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/3459746896 shutdown_connections 2023-12-13T23:56:13.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.073+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/3459746896 >> 172.21.15.124:0/3459746896 conn(0x7f0a3806a4d0 msgr2=0x7f0a3806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:13.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.074+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/3459746896 shutdown_connections 2023-12-13T23:56:13.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.074+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/3459746896 wait complete. 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 Processor -- start 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 -- start start 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a24003ef0 0x7f0a380ab8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 0x7f0a380aa730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a380c4990 con 0x7f0a381599b0 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38131e90 con 0x7f0a380c4510 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3f9ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a38131280 con 0x7f0a24003ef0 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.075+0000 7f0a3cf69640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42940/0 (socket says 172.21.15.124:42940) 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 -- 172.21.15.124:0/4290848957 learned_addr learned my addr 172.21.15.124:0/4290848957 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3df6b640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 0x7f0a380aa730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 -- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a24003ef0 msgr2=0x7f0a380ab8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a24003ef0 0x7f0a380ab8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 -- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 msgr2=0x7f0a380aa730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 0x7f0a380aa730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 -- 172.21.15.124:0/4290848957 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38131560 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3df6b640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 0x7f0a380aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3cf69640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0a2802c2b0 tx=0x7f0a28004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28003810 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a28003970 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a38131840 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a28003d20 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.076+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a38159e10 con 0x7f0a381599b0 2023-12-13T23:56:13.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.077+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a380722f0 con 0x7f0a381599b0 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.078+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0a28034440 con 0x7f0a381599b0 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.078+0000 7f0a2dffb640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 0x7f0a1c06d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.078+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a280ab5e0 con 0x7f0a381599b0 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.079+0000 7f0a3d76a640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 0x7f0a1c06d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.079+0000 7f0a3d76a640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 0x7f0a1c06d670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0a34006e80 tx=0x7f0a34006df0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.083+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0a28076710 con 0x7f0a381599b0 2023-12-13T23:56:13.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.313+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f0a38069400 con 0x7f0a381599b0 2023-12-13T23:56:13.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.313+0000 7f0a2dffb640 1 -- 172.21.15.124:0/4290848957 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f0a28079d30 con 0x7f0a381599b0 2023-12-13T23:56:13.312 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "auth_dump": [ 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.a", 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplFo6KDBAArJJKNRTjxYdaYwtbHVbu7g==", 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.313 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.b", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpl893rGBAAas73Q+Mux4HJCWgSronQlQ==", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.c", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpllD7vExAAj4trq2lGnNXIIi2G5k+/IQ==", 2023-12-13T23:56:13.314 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.d", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpliKtJIBAAz3MbEgTL5bv/l2zLj32n5w==", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:56:13.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.0", 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAqRHplcCLWBxAAxYxXKFh6L9WK7uOg5UqRLA==", 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.316 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.1", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAtRHplDW73BhAAQoBtr86nfF318aMqfI/7tA==", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.317 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.2", 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAwRHpl+Bi6BBAAR9+uNWYlp4BCI3gmEbvp3Q==", 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.3", 2023-12-13T23:56:13.318 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAzRHplrYhsAhAADvJtD6bIDoXVG50rEYkYaQ==", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.4", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA5RHpleClsCxAAeVn2LqqB2gIUnF3HAimTnw==", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.319 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.5", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA8RHpl552xCRAAgCBrpQroUXTlg+wox8Fx/w==", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.6", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA/RHplFgN+BxAAkUw1AQX9IJpOvNgPFBkQ3g==", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.7", 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBCRHplw4NRBRAAbPuwtmaK8/9lrd4crZHHlg==", 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.321 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.0", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw==", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.1", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplz1vULRAANGEKQl90MW0+VmH0N96EXw==", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:56:13.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.2", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplGAlqKhAA3u5mk3DyH7KsWsAScLW0Ew==", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.323 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.3", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpl7tuiMxAAF4ySYml4t9rxEC9Ji3/wdw==", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.admin", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAlRHpl0JrlKxAAZUPeNVBB/yhD505NEeuUow==", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:56:13.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-mds", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHplnvvqOhAAWjuoph9SyrJlIs0H1/uKFg==", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-mds" 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-mgr", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHpl+QPrOhAAp1oyZOdvvYKFHjGag1rzJw==", 2023-12-13T23:56:13.325 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-mgr" 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-osd", 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHpl6QrrOhAATlAPLgyOEFaKDqMLPfaerA==", 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-osd" 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-rbd", 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHplhhHrOhAAcnPO+2m3DlXA5LlWmnM0xA==", 2023-12-13T23:56:13.326 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-rbd" 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHplaBjrOhAAFwQ66z3tutiEzjzmq3TjAw==", 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.bootstrap-rgw", 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBHRHplCB/rOhAASjxpqiYf19LgMTHFyz72/g==", 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.327 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile bootstrap-rgw" 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mgr.x", 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplEAcvCRAAuDLwmskaKHqzdX0/vTGodQ==", 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile mgr", 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mgr.y", 2023-12-13T23:56:13.328 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplDCGRBRAATplO/LAqiqpuOu7VOMvxog==", 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile mgr", 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout: ] 2023-12-13T23:56:13.329 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:56:13.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 msgr2=0x7f0a1c06d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 0x7f0a1c06d670 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0a34006e80 tx=0x7f0a34006df0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 msgr2=0x7f0a380c3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0a2802c2b0 tx=0x7f0a28004730 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 shutdown_connections 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a1c06b1b0 0x7f0a1c06d670 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a380c4510 0x7f0a380aa730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a381599b0 0x7f0a380c3fd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 --2- 172.21.15.124:0/4290848957 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a24003ef0 0x7f0a380ab8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 >> 172.21.15.124:0/4290848957 conn(0x7f0a3806a4d0 msgr2=0x7f0a380aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:13.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 shutdown_connections 2023-12-13T23:56:13.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.315+0000 7f0a3f9ce640 1 -- 172.21.15.124:0/4290848957 wait complete. 2023-12-13T23:56:13.333 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-12-13T23:56:13.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.435+0000 7f6f837c9640 1 Processor -- start 2023-12-13T23:56:13.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.435+0000 7f6f837c9640 1 -- start start 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f837c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f837c9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6f7c15abf0 0x7f6f7c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f837c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c15e2a0 con 0x7f6f7c15e730 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f837c9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7c15b520 con 0x7f6f7c15abf0 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f837c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c143870 con 0x7f6f7c143490 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6f7c15abf0 0x7f6f7c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6f7c15abf0 0x7f6f7c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42954/0 (socket says 172.21.15.124:42954) 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 -- 172.21.15.124:0/3332272319 learned_addr learned my addr 172.21.15.124:0/3332272319 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 -- 172.21.15.124:0/3332272319 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 msgr2=0x7f6f7c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 --2- 172.21.15.124:0/3332272319 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 -- 172.21.15.124:0/3332272319 >> v1:172.21.15.203:6789/0 conn(0x7f6f7c15e730 legacy=0x7f6f7c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.436+0000 7f6f81d66640 1 -- 172.21.15.124:0/3332272319 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c15b240 con 0x7f6f7c15abf0 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f81d66640 1 --2- 172.21.15.124:0/3332272319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c0b1230 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009e50 tx=0x7f6f7402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d699d420c00dc8c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f73fff640 1 -- 172.21.15.124:0/3332272319 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7402ec80 con 0x7f6f7c15abf0 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f73fff640 1 -- 172.21.15.124:0/3332272319 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7402ede0 con 0x7f6f7c15abf0 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f73fff640 1 -- 172.21.15.124:0/3332272319 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74036610 con 0x7f6f7c15abf0 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f837c9640 1 -- 172.21.15.124:0/3332272319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 msgr2=0x7f6f7c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.437+0000 7f6f837c9640 1 --2- 172.21.15.124:0/3332272319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c0b1230 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6f74009e50 tx=0x7f6f7402e0e0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 -- 172.21.15.124:0/3332272319 shutdown_connections 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 --2- 172.21.15.124:0/3332272319 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c0b1230 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 --2- 172.21.15.124:0/3332272319 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 -- 172.21.15.124:0/3332272319 >> 172.21.15.124:0/3332272319 conn(0x7f6f7c06a4d0 msgr2=0x7f6f7c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:13.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 -- 172.21.15.124:0/3332272319 shutdown_connections 2023-12-13T23:56:13.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.438+0000 7f6f837c9640 1 -- 172.21.15.124:0/3332272319 wait complete. 2023-12-13T23:56:13.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 Processor -- start 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 -- start start 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f7c15e730 0x7f6f7c13e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c13f7a0 con 0x7f6f7c15abf0 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c13cf00 con 0x7f6f7c15e730 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.439+0000 7f6f837c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7c13d1e0 con 0x7f6f7c143490 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42970/0 (socket says 172.21.15.124:42970) 2023-12-13T23:56:13.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 -- 172.21.15.124:0/2613840341 learned_addr learned my addr 172.21.15.124:0/2613840341 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f81d66640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f7c15e730 0x7f6f7c13e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f81565640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 -- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 msgr2=0x7f6f7c159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 -- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f7c15e730 msgr2=0x7f6f7c13e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f7c15e730 0x7f6f7c13e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 -- 172.21.15.124:0/2613840341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f74009b30 con 0x7f6f7c15abf0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f80d64640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00b3e0 tx=0x7f6f6c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011830 con 0x7f6f7c15abf0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c0119b0 con 0x7f6f7c15abf0 2023-12-13T23:56:13.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c011d60 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c13d4c0 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.440+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c159b30 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.441+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f7c159d80 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.442+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6f6c01a400 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.442+0000 7f6f71ffb640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 0x7f6f6006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.442+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6f6c089890 con 0x7f6f7c15abf0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.442+0000 7f6f81565640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 0x7f6f6006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.443+0000 7f6f81565640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 0x7f6f6006d680 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c15cdb0 tx=0x7f6f78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.446+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6f6c054910 con 0x7f6f7c15abf0 2023-12-13T23:56:13.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.679+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f6f7c13d6b0 con 0x7f6f7c15abf0 2023-12-13T23:56:13.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.681+0000 7f6f71ffb640 1 -- 172.21.15.124:0/2613840341 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f6f6c015020 con 0x7f6f7c15abf0 2023-12-13T23:56:13.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 msgr2=0x7f6f6006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 0x7f6f6006d680 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c15cdb0 tx=0x7f6f78008040 comp rx=0 tx=0).stop 2023-12-13T23:56:13.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 msgr2=0x7f6f7c158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00b3e0 tx=0x7f6f6c00b8a0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 shutdown_connections 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6f6006b1c0 0x7f6f6006d680 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6f7c15e730 0x7f6f7c13e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f7c15abf0 0x7f6f7c158c00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 --2- 172.21.15.124:0/2613840341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6f7c143490 0x7f6f7c159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 >> 172.21.15.124:0/2613840341 conn(0x7f6f7c06a4d0 msgr2=0x7f6f7c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 shutdown_connections 2023-12-13T23:56:13.683 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.683+0000 7f6f837c9640 1 -- 172.21.15.124:0/2613840341 wait complete. 2023-12-13T23:56:13.693 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-12-13T23:56:13.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.794+0000 7eff58753640 1 Processor -- start 2023-12-13T23:56:13.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.794+0000 7eff58753640 1 -- start start 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff58753640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff500b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff58753640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff50071b40 con 0x7eff50144a40 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff58753640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff50144e20 con 0x7eff500702e0 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff58753640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff500ccc00 con 0x7eff500b4350 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff564ef640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7eff50144a40 0x7eff50071fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57832/0 (socket says 172.21.15.124:57832) 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff564ef640 1 -- 172.21.15.124:0/3395160816 learned_addr learned my addr 172.21.15.124:0/3395160816 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:13.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023487596 0 0) 0x7eff50071b40 con 0x7eff50144a40 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff3c003520 con 0x7eff50144a40 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff55cee640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff500b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3763368186 0 0) 0x7eff3c003520 con 0x7eff50144a40 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 msgr2=0x7eff500b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff500b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 >> v1:172.21.15.203:6789/0 conn(0x7eff500702e0 legacy=0x7eff50070770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff50070f30 con 0x7eff50144a40 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7eff4c001670 con 0x7eff50144a40 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.795+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 >> v1:172.21.15.124:6789/0 conn(0x7eff50144a40 legacy=0x7eff50071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff500ccc00 con 0x7eff3c0039a0 2023-12-13T23:56:13.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c00bb60 con 0x7eff3c004d00 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c00bd20 con 0x7eff500b4350 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff56cf0640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff55cee640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 -- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 msgr2=0x7eff3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff500b4350 0x7eff3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 -- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c004d00 msgr2=0x7eff3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c004d00 0x7eff3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.796+0000 7eff564ef640 1 -- 172.21.15.124:0/3395160816 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c003520 con 0x7eff3c0039a0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff564ef640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff3c003dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00c240 tx=0x7eff3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c63edb1179ff68 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c005ad0 con 0x7eff3c0039a0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c005c30 con 0x7eff3c0039a0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff554ed640 1 -- 172.21.15.124:0/3395160816 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c02c8b0 con 0x7eff3c0039a0 2023-12-13T23:56:13.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 -- 172.21.15.124:0/3395160816 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 msgr2=0x7eff3c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 --2- 172.21.15.124:0/3395160816 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff3c003dc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7eff4c00c240 tx=0x7eff3c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff56cf0640 1 -- 172.21.15.124:0/3395160816 reap_dead start 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 -- 172.21.15.124:0/3395160816 shutdown_connections 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 -- 172.21.15.124:0/3395160816 >> 172.21.15.124:0/3395160816 conn(0x7eff5006c520 msgr2=0x7eff5006c900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 -- 172.21.15.124:0/3395160816 shutdown_connections 2023-12-13T23:56:13.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.797+0000 7eff58753640 1 -- 172.21.15.124:0/3395160816 wait complete. 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.798+0000 7eff58753640 1 Processor -- start 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.798+0000 7eff58753640 1 -- start start 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff501e1110 0x7eff501e1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff501e4de0 0x7eff500ac560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5012d660 con 0x7eff3c0039a0 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5012d940 con 0x7eff501e4de0 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff58753640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5012dc20 con 0x7eff501e1110 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42982/0 (socket says 172.21.15.124:42982) 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 -- 172.21.15.124:0/2117612978 learned_addr learned my addr 172.21.15.124:0/2117612978 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 -- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff501e1110 msgr2=0x7eff501e1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff501e1110 0x7eff501e1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 -- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff501e4de0 msgr2=0x7eff500ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff501e4de0 0x7eff500ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.799+0000 7eff564ef640 1 -- 172.21.15.124:0/2117612978 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff5012df00 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff564ef640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7eff4c001930 tx=0x7eff4c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c039070 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4c02ccc0 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff5012e1e0 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff4c03c040 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.800+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff501e5440 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.801+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff5012da80 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.801+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7eff4c032070 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.801+0000 7eff46ffd640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 0x7eff3406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.802+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7eff4c0aa9f0 con 0x7eff3c0039a0 2023-12-13T23:56:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.802+0000 7eff55cee640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 0x7eff3406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:13.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.802+0000 7eff55cee640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 0x7eff3406d680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff40006f00 tx=0x7eff40006e90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:13.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:13.806+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7eff5012da80 con 0x7eff3c0039a0 2023-12-13T23:56:14.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.040+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7eff5012dd60 con 0x7eff3c0039a0 2023-12-13T23:56:14.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.042+0000 7eff46ffd640 1 -- 172.21.15.124:0/2117612978 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7eff5012dd60 con 0x7eff3c0039a0 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 msgr2=0x7eff3406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 0x7eff3406d680 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7eff40006f00 tx=0x7eff40006e90 comp rx=0 tx=0).stop 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 msgr2=0x7eff5015aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7eff4c001930 tx=0x7eff4c001a00 comp rx=0 tx=0).stop 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 shutdown_connections 2023-12-13T23:56:14.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff3406b1c0 0x7eff3406d680 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff501e4de0 0x7eff500ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff501e1110 0x7eff501e1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 --2- 172.21.15.124:0/2117612978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0039a0 0x7eff5015aea0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 >> 172.21.15.124:0/2117612978 conn(0x7eff5006c520 msgr2=0x7eff500ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 shutdown_connections 2023-12-13T23:56:14.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.044+0000 7eff58753640 1 -- 172.21.15.124:0/2117612978 wait complete. 2023-12-13T23:56:14.054 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-12-13T23:56:14.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.155+0000 7fe620ff1640 1 Processor -- start 2023-12-13T23:56:14.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.155+0000 7fe620ff1640 1 -- start start 2023-12-13T23:56:14.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe61c15d1d0 0x7fe61c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe61c0ad640 0x7fe61c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c0aeb80 0x7fe61c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61c0adae0 con 0x7fe61c0ad640 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61c0addc0 con 0x7fe61c15d1d0 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe620ff1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c0ae0a0 con 0x7fe61c0aeb80 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe61c0ad640 0x7fe61c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe61c0ad640 0x7fe61c06ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:42986/0 (socket says 172.21.15.124:42986) 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 -- 172.21.15.124:0/793227809 learned_addr learned my addr 172.21.15.124:0/793227809 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe61ad9d640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c0aeb80 0x7fe61c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe61a59c640 1 --2- 172.21.15.124:0/793227809 >> v2:172.21.15.203:3300/0 conn(0x7fe61c15d1d0 0x7fe61c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 -- 172.21.15.124:0/793227809 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c0aeb80 msgr2=0x7fe61c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c0aeb80 0x7fe61c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 -- 172.21.15.124:0/793227809 >> v2:172.21.15.203:3300/0 conn(0x7fe61c15d1d0 msgr2=0x7fe61c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:14.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 --2- 172.21.15.124:0/793227809 >> v2:172.21.15.203:3300/0 conn(0x7fe61c15d1d0 0x7fe61c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.156+0000 7fe613fff640 1 -- 172.21.15.124:0/793227809 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe61c0ae380 con 0x7fe61c0ad640 2023-12-13T23:56:14.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe613fff640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c06ba20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe61c072310 tx=0x7fe60402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b39eb92ea4a71860 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe619d9b640 1 -- 172.21.15.124:0/793227809 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604031870 con 0x7fe61c0ad640 2023-12-13T23:56:14.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe619d9b640 1 -- 172.21.15.124:0/793227809 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6040319d0 con 0x7fe61c0ad640 2023-12-13T23:56:14.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe619d9b640 1 -- 172.21.15.124:0/793227809 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604031d60 con 0x7fe61c0ad640 2023-12-13T23:56:14.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe620ff1640 1 -- 172.21.15.124:0/793227809 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 msgr2=0x7fe61c06ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe620ff1640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c06ba20 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fe61c072310 tx=0x7fe60402b910 comp rx=0 tx=0).stop 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe620ff1640 1 -- 172.21.15.124:0/793227809 shutdown_connections 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.157+0000 7fe620ff1640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c06ba20 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.158+0000 7fe620ff1640 1 --2- 172.21.15.124:0/793227809 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c0aeb80 0x7fe61c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.158+0000 7fe620ff1640 1 --2- 172.21.15.124:0/793227809 >> v2:172.21.15.203:3300/0 conn(0x7fe61c15d1d0 0x7fe61c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.158+0000 7fe620ff1640 1 -- 172.21.15.124:0/793227809 >> 172.21.15.124:0/793227809 conn(0x7fe61c070e70 msgr2=0x7fe61c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.158+0000 7fe620ff1640 1 -- 172.21.15.124:0/793227809 shutdown_connections 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.158+0000 7fe620ff1640 1 -- 172.21.15.124:0/793227809 wait complete. 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.159+0000 7fe620ff1640 1 Processor -- start 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.159+0000 7fe620ff1640 1 -- start start 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe61c0aeb80 0x7fe61c14e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 0x7fe61c1f3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c1f3cd0 con 0x7fe61c0ad640 2023-12-13T23:56:14.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c14f8b0 con 0x7fe61c0aeb80 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61c14eca0 con 0x7fe61c15d1d0 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43000/0 (socket says 172.21.15.124:43000) 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 -- 172.21.15.124:0/1465216569 learned_addr learned my addr 172.21.15.124:0/1465216569 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe613fff640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe61c0aeb80 0x7fe61c14e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61ad9d640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 0x7fe61c1f3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 -- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 msgr2=0x7fe61c1f3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 0x7fe61c1f3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 -- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe61c0aeb80 msgr2=0x7fe61c14e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe61c0aeb80 0x7fe61c14e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 -- 172.21.15.124:0/1465216569 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6040094d0 con 0x7fe61c0ad640 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61ad9d640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 0x7fe61c1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe61a59c640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00d5f0 tx=0x7fe60c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c0125f0 con 0x7fe61c0ad640 2023-12-13T23:56:14.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c012750 con 0x7fe61c0ad640 2023-12-13T23:56:14.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe61c14ef80 con 0x7fe61c0ad640 2023-12-13T23:56:14.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c012b00 con 0x7fe61c0ad640 2023-12-13T23:56:14.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.160+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe61c14f540 con 0x7fe61c0ad640 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.161+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe61c069400 con 0x7fe61c0ad640 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.162+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe60c012c60 con 0x7fe61c0ad640 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.162+0000 7fe612ffd640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 0x7fe5fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.162+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe60c089a60 con 0x7fe61c0ad640 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.162+0000 7fe613fff640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 0x7fe5fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.163+0000 7fe613fff640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 0x7fe5fc06d680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe604036000 tx=0x7fe60403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.166+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe60c054b70 con 0x7fe61c0ad640 2023-12-13T23:56:14.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.395+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7fe61c06b2f0 con 0x7fe61c0ad640 2023-12-13T23:56:14.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.397+0000 7fe612ffd640 1 -- 172.21.15.124:0/1465216569 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7fe60c058190 con 0x7fe61c0ad640 2023-12-13T23:56:14.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 msgr2=0x7fe5fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 0x7fe5fc06d680 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe604036000 tx=0x7fe60403c040 comp rx=0 tx=0).stop 2023-12-13T23:56:14.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 msgr2=0x7fe61c1f5cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe60c00d5f0 tx=0x7fe60c00dab0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 shutdown_connections 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5fc06b1c0 0x7fe5fc06d680 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe61c0aeb80 0x7fe61c14e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe61c15d1d0 0x7fe61c1f3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 --2- 172.21.15.124:0/1465216569 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe61c0ad640 0x7fe61c1f5cd0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 >> 172.21.15.124:0/1465216569 conn(0x7fe61c070e70 msgr2=0x7fe61c06b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 shutdown_connections 2023-12-13T23:56:14.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.399+0000 7fe620ff1640 1 -- 172.21.15.124:0/1465216569 wait complete. 2023-12-13T23:56:14.410 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-12-13T23:56:14.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.513+0000 7ff23fb7c640 1 Processor -- start 2023-12-13T23:56:14.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.513+0000 7ff23fb7c640 1 -- start start 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23fb7c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff238158ba0 0x7ff23815c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23fb7c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff2380ac370 0x7ff2380ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23fb7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2380adbb0 con 0x7ff238071c80 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23fb7c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2380b4520 con 0x7ff2380ac370 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23fb7c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff23815c6a0 con 0x7ff238158ba0 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff2380ac370 0x7ff2380ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff2380ac370 0x7ff2380ae5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43008/0 (socket says 172.21.15.124:43008) 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 -- 172.21.15.124:0/1682704458 learned_addr learned my addr 172.21.15.124:0/1682704458 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d918640 1 --2- 172.21.15.124:0/1682704458 >> v2:172.21.15.203:3300/0 conn(0x7ff238158ba0 0x7ff23815c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 -- 172.21.15.124:0/1682704458 >> v2:172.21.15.203:3300/0 conn(0x7ff238158ba0 msgr2=0x7ff23815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:14.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 --2- 172.21.15.124:0/1682704458 >> v2:172.21.15.203:3300/0 conn(0x7ff238158ba0 0x7ff23815c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 -- 172.21.15.124:0/1682704458 >> v1:172.21.15.203:6789/0 conn(0x7ff238071c80 legacy=0x7ff2380b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:14.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.514+0000 7ff23d117640 1 -- 172.21.15.124:0/1682704458 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2380b3f60 con 0x7ff2380ac370 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23d117640 1 --2- 172.21.15.124:0/1682704458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2380ae5e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff238073da0 tx=0x7ff22802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bd51be018df5531 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23c916640 1 -- 172.21.15.124:0/1682704458 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2280317c0 con 0x7ff2380ac370 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23c916640 1 -- 172.21.15.124:0/1682704458 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228031920 con 0x7ff2380ac370 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23c916640 1 -- 172.21.15.124:0/1682704458 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228031cb0 con 0x7ff2380ac370 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1682704458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 msgr2=0x7ff2380ae5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1682704458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2380ae5e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7ff238073da0 tx=0x7ff22802b860 comp rx=0 tx=0).stop 2023-12-13T23:56:14.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.515+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1682704458 shutdown_connections 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.516+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1682704458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2380ae5e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.516+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1682704458 >> v2:172.21.15.203:3300/0 conn(0x7ff238158ba0 0x7ff23815c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.516+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1682704458 >> 172.21.15.124:0/1682704458 conn(0x7ff2380b4720 msgr2=0x7ff2380b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.516+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1682704458 shutdown_connections 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.516+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1682704458 wait complete. 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 Processor -- start 2023-12-13T23:56:14.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 -- start start 2023-12-13T23:56:14.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff238071c80 0x7ff2381447c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff238158ba0 0x7ff2381f6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2381e6bf0 con 0x7ff2380ac370 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2381e4bb0 con 0x7ff238158ba0 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.517+0000 7ff23fb7c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2381e4e90 con 0x7ff238071c80 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43012/0 (socket says 172.21.15.124:43012) 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 -- 172.21.15.124:0/1840117552 learned_addr learned my addr 172.21.15.124:0/1840117552 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23e119640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff238158ba0 0x7ff2381f6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d918640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff238071c80 0x7ff2381447c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 -- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff238071c80 msgr2=0x7ff2381447c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff238071c80 0x7ff2381447c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 -- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff238158ba0 msgr2=0x7ff2381f6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff238158ba0 0x7ff2381f6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 -- 172.21.15.124:0/1840117552 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280094d0 con 0x7ff2380ac370 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23d117640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff22802b8a0 tx=0x7ff2280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228032860 con 0x7ff2380ac370 2023-12-13T23:56:14.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2280329e0 con 0x7ff2380ac370 2023-12-13T23:56:14.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2381e5170 con 0x7ff2380ac370 2023-12-13T23:56:14.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228032d90 con 0x7ff2380ac370 2023-12-13T23:56:14.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.518+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2381e6810 con 0x7ff2380ac370 2023-12-13T23:56:14.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.519+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff238069400 con 0x7ff2380ac370 2023-12-13T23:56:14.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.520+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff22804e450 con 0x7ff2380ac370 2023-12-13T23:56:14.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.520+0000 7ff22e7fc640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 0x7ff21c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.520+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2280ab720 con 0x7ff2380ac370 2023-12-13T23:56:14.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.520+0000 7ff23d918640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 0x7ff21c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.521+0000 7ff23d918640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 0x7ff21c06d680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.522 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.524+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff228076850 con 0x7ff2380ac370 2023-12-13T23:56:14.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.754+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7ff238070b70 con 0x7ff2380ac370 2023-12-13T23:56:14.755 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.757+0000 7ff22e7fc640 1 -- 172.21.15.124:0/1840117552 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff228079e70 con 0x7ff2380ac370 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 msgr2=0x7ff21c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 0x7ff21c06d680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7ff234006fd0 tx=0x7ff234008040 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 msgr2=0x7ff2381f6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff22802b8a0 tx=0x7ff2280046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 shutdown_connections 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff21c06b1c0 0x7ff21c06d680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff238158ba0 0x7ff2381f6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff2380ac370 0x7ff2381f6830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 --2- 172.21.15.124:0/1840117552 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff238071c80 0x7ff2381447c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 >> 172.21.15.124:0/1840117552 conn(0x7ff2380b4720 msgr2=0x7ff238159b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 shutdown_connections 2023-12-13T23:56:14.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.759+0000 7ff23fb7c640 1 -- 172.21.15.124:0/1840117552 wait complete. 2023-12-13T23:56:14.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-12-13T23:56:14.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 Processor -- start 2023-12-13T23:56:14.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 -- start start 2023-12-13T23:56:14.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1ac415d1d0 0x7f1ac406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac40ad640 0x7f1ac406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40adae0 con 0x7f1ac40aeb80 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.871+0000 7f1ac8d3c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40addc0 con 0x7f1ac415d1d0 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac8d3c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ae0a0 con 0x7f1ac40ad640 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1ac415d1d0 0x7f1ac406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1ac415d1d0 0x7f1ac406b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43028/0 (socket says 172.21.15.124:43028) 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 -- 172.21.15.124:0/166095201 learned_addr learned my addr 172.21.15.124:0/166095201 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1abbfff640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac40ad640 0x7f1ac406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 -- 172.21.15.124:0/166095201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac40ad640 msgr2=0x7f1ac406ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac40ad640 0x7f1ac406ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 -- 172.21.15.124:0/166095201 >> v1:172.21.15.203:6789/0 conn(0x7f1ac40aeb80 legacy=0x7f1ac4160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.872+0000 7f1ac259c640 1 -- 172.21.15.124:0/166095201 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40ae380 con 0x7f1ac415d1d0 2023-12-13T23:56:14.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac259c640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac415d1d0 0x7f1ac406b4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009870 tx=0x7f1ab402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f33fd7a5fbd213a2 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac1d9b640 1 -- 172.21.15.124:0/166095201 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030840 con 0x7f1ac415d1d0 2023-12-13T23:56:14.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac1d9b640 1 -- 172.21.15.124:0/166095201 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab40309a0 con 0x7f1ac415d1d0 2023-12-13T23:56:14.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac1d9b640 1 -- 172.21.15.124:0/166095201 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030d30 con 0x7f1ac415d1d0 2023-12-13T23:56:14.877 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/166095201 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac415d1d0 msgr2=0x7f1ac406b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac415d1d0 0x7f1ac406b4e0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4009870 tx=0x7f1ab402b980 comp rx=0 tx=0).stop 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/166095201 shutdown_connections 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac40ad640 0x7f1ac406ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/166095201 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac415d1d0 0x7f1ac406b4e0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/166095201 >> 172.21.15.124:0/166095201 conn(0x7f1ac4070e70 msgr2=0x7f1ac4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/166095201 shutdown_connections 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.873+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/166095201 wait complete. 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 Processor -- start 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 -- start start 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac40aeb80 0x7f1ac41f6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 0x7f1ac41f6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ce580 con 0x7f1ac40ad640 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ce860 con 0x7f1ac40aeb80 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.875+0000 7f1ac8d3c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ceb40 con 0x7f1ac415d1d0 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43034/0 (socket says 172.21.15.124:43034) 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 -- 172.21.15.124:0/2337809108 learned_addr learned my addr 172.21.15.124:0/2337809108 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1abbfff640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac40aeb80 0x7f1ac41f6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac2d9d640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 0x7f1ac41f6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 -- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 msgr2=0x7f1ac41f6f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 0x7f1ac41f6f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 -- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac40aeb80 msgr2=0x7f1ac41f6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac40aeb80 0x7f1ac41f6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 -- 172.21.15.124:0/2337809108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab40094d0 con 0x7f1ac40ad640 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac2d9d640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 0x7f1ac41f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac259c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4000c00 tx=0x7f1ab4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.879 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4039a90 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab4039bf0 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac40cee20 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4043410 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.876+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40cd8c0 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.877+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac4069400 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.878+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1ab404c3b0 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.878+0000 7f1abaffd640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 0x7f1aa406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.878+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1ab40aba80 con 0x7f1ac40ad640 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.878+0000 7f1abbfff640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 0x7f1aa406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:14.880 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.879+0000 7f1abbfff640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 0x7f1aa406d680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4071500 tx=0x7f1aac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:14.881 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:14.882+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1ab4076c40 con 0x7f1ac40ad640 2023-12-13T23:56:15.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.118+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f1ac4071530 con 0x7f1ac40ad640 2023-12-13T23:56:15.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.120+0000 7f1abaffd640 1 -- 172.21.15.124:0/2337809108 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ab407a260 con 0x7f1ac40ad640 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 msgr2=0x7f1aa406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 0x7f1aa406d680 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4071500 tx=0x7f1aac008040 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 msgr2=0x7f1ac4150000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1ab4000c00 tx=0x7f1ab4004730 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 shutdown_connections 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1aa406b1c0 0x7f1aa406d680 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1ac40aeb80 0x7f1ac41f6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1ac415d1d0 0x7f1ac41f6f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 --2- 172.21.15.124:0/2337809108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ac40ad640 0x7f1ac4150000 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 >> 172.21.15.124:0/2337809108 conn(0x7f1ac4070e70 msgr2=0x7f1ac41528c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 shutdown_connections 2023-12-13T23:56:15.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.122+0000 7f1ac8d3c640 1 -- 172.21.15.124:0/2337809108 wait complete. 2023-12-13T23:56:15.131 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-12-13T23:56:15.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.232+0000 7fba6d197640 1 Processor -- start 2023-12-13T23:56:15.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.232+0000 7fba6d197640 1 -- start start 2023-12-13T23:56:15.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6d197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba680b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6d197640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6815e2a0 con 0x7fba6815e730 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6d197640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba6815b240 con 0x7fba6815abf0 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6d197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6815b520 con 0x7fba68143490 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6659c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fba6815e730 0x7fba6815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57848/0 (socket says 172.21.15.124:57848) 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba6659c640 1 -- 172.21.15.124:0/1204322105 learned_addr learned my addr 172.21.15.124:0/1204322105 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba66d9d640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba680b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224977752 0 0) 0x7fba6815e2a0 con 0x7fba6815e730 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50003520 con 0x7fba6815e730 2023-12-13T23:56:15.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242992662 0 0) 0x7fba50003520 con 0x7fba6815e730 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 msgr2=0x7fba680b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba680b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 >> v1:172.21.15.203:6789/0 conn(0x7fba6815abf0 legacy=0x7fba680b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6815dc50 con 0x7fba6815e730 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fba5c001670 con 0x7fba6815e730 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 >> v1:172.21.15.124:6789/0 conn(0x7fba6815e730 legacy=0x7fba6815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.233+0000 7fba65d9b640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba65d9b640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba50004d00 0x7fba50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6815b520 con 0x7fba500039a0 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5000dc40 con 0x7fba50004d00 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5000de00 con 0x7fba68143490 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.242 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba6659c640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba6759e640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba50004d00 0x7fba50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 -- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 msgr2=0x7fba500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba68143490 0x7fba500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 -- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba50004d00 msgr2=0x7fba50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba50004d00 0x7fba50005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.234+0000 7fba66d9d640 1 -- 172.21.15.124:0/1204322105 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba50003520 con 0x7fba500039a0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba66d9d640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba50003dc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fba60009a90 tx=0x7fba500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d94a8a73f46e753 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6002c680 con 0x7fba500039a0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba6002c7e0 con 0x7fba500039a0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba65d9b640 1 -- 172.21.15.124:0/1204322105 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6002cb70 con 0x7fba500039a0 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 -- 172.21.15.124:0/1204322105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 msgr2=0x7fba50003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 --2- 172.21.15.124:0/1204322105 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba50003dc0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fba60009a90 tx=0x7fba500057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6759e640 1 -- 172.21.15.124:0/1204322105 reap_dead start 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 -- 172.21.15.124:0/1204322105 shutdown_connections 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 -- 172.21.15.124:0/1204322105 >> 172.21.15.124:0/1204322105 conn(0x7fba6806a4d0 msgr2=0x7fba6806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 -- 172.21.15.124:0/1204322105 shutdown_connections 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.235+0000 7fba6d197640 1 -- 172.21.15.124:0/1204322105 wait complete. 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 Processor -- start 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 -- start start 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba6813f2c0 0x7fba6813fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba68159910 0x7fba68070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68140fb0 con 0x7fba500039a0 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68140330 con 0x7fba68159910 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba6d197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba68140610 con 0x7fba6813f2c0 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.237+0000 7fba66d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43044/0 (socket says 172.21.15.124:43044) 2023-12-13T23:56:15.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 -- 172.21.15.124:0/368126380 learned_addr learned my addr 172.21.15.124:0/368126380 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba6659c640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba6813f2c0 0x7fba6813fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba6759e640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba68159910 0x7fba68070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 -- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba6813f2c0 msgr2=0x7fba6813fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba6813f2c0 0x7fba6813fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 -- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba68159910 msgr2=0x7fba68070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba68159910 0x7fba68070dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 -- 172.21.15.124:0/368126380 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba681408f0 con 0x7fba500039a0 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba66d9d640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003810 con 0x7fba500039a0 2023-12-13T23:56:15.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba60003970 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba68140bd0 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba60003d20 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.238+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6813ff90 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.239+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba68069400 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.240+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fba60034440 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.240+0000 7fba4f7fe640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 0x7fba5406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.240+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fba600ab7b0 con 0x7fba500039a0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.240+0000 7fba6659c640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 0x7fba5406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.241+0000 7fba6659c640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 0x7fba5406d680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba680ae3e0 tx=0x7fba5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.244+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fba60076a00 con 0x7fba500039a0 2023-12-13T23:56:15.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.475+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7fba680b2820 con 0x7fba500039a0 2023-12-13T23:56:15.476 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.477+0000 7fba4f7fe640 1 -- 172.21.15.124:0/368126380 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7fba600044e0 con 0x7fba500039a0 2023-12-13T23:56:15.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 msgr2=0x7fba5406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 0x7fba5406d680 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fba680ae3e0 tx=0x7fba5c009040 comp rx=0 tx=0).stop 2023-12-13T23:56:15.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 msgr2=0x7fba681413d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fba60000c00 tx=0x7fba60004730 comp rx=0 tx=0).stop 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 shutdown_connections 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba5406b1c0 0x7fba5406d680 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba68159910 0x7fba68070dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba6813f2c0 0x7fba6813fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 --2- 172.21.15.124:0/368126380 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba500039a0 0x7fba681413d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 >> 172.21.15.124:0/368126380 conn(0x7fba6806a4d0 msgr2=0x7fba6815c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 shutdown_connections 2023-12-13T23:56:15.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.479+0000 7fba6d197640 1 -- 172.21.15.124:0/368126380 wait complete. 2023-12-13T23:56:15.489 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:15.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.589+0000 7fa4c0cc9640 1 Processor -- start 2023-12-13T23:56:15.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.589+0000 7fa4c0cc9640 1 -- start start 2023-12-13T23:56:15.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4c0cc9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4c0cc9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa4bc0b4370 0x7fa4bc0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4c0cc9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4bc071770 con 0x7fa4bc0ceb70 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4c0cc9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4bc0b4d20 con 0x7fa4bc0b4370 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4c0cc9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc0ccb60 con 0x7fa4bc144980 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4ba59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.590+0000 7fa4bad9d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa4bc0b4370 0x7fa4bc0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4ba59c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:39172/0 (socket says 172.21.15.124:39172) 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4b9d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa4bc0ceb70 0x7fa4bc0cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:42142/0 (socket says 172.21.15.124:42142) 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4ba59c640 1 -- 172.21.15.124:0/1866316038 learned_addr learned my addr 172.21.15.124:0/1866316038 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4bad9d640 1 -- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 msgr2=0x7fa4bc15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4bad9d640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4bad9d640 1 -- 172.21.15.124:0/1866316038 >> v1:172.21.15.203:6789/0 conn(0x7fa4bc0ceb70 legacy=0x7fa4bc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4bad9d640 1 -- 172.21.15.124:0/1866316038 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4bc0ccdc0 con 0x7fa4bc0b4370 2023-12-13T23:56:15.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.591+0000 7fa4ba59c640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:15.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4bad9d640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc0b47e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40097a0 tx=0x7fa4b402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10be8c72d8f3c512 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4b959a640 1 -- 172.21.15.124:0/1866316038 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4031830 con 0x7fa4bc0b4370 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4b959a640 1 -- 172.21.15.124:0/1866316038 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4b4031990 con 0x7fa4bc0b4370 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4b959a640 1 -- 172.21.15.124:0/1866316038 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4b4031d20 con 0x7fa4bc0b4370 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc0b4370 msgr2=0x7fa4bc0b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc0b47e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa4b40097a0 tx=0x7fa4b402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1866316038 shutdown_connections 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc0b47e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1866316038 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc144980 0x7fa4bc15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1866316038 >> 172.21.15.124:0/1866316038 conn(0x7fa4bc06a4d0 msgr2=0x7fa4bc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.592+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1866316038 shutdown_connections 2023-12-13T23:56:15.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.593+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1866316038 wait complete. 2023-12-13T23:56:15.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.594+0000 7fa4c0cc9640 1 Processor -- start 2023-12-13T23:56:15.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.594+0000 7fa4c0cc9640 1 -- start start 2023-12-13T23:56:15.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc0ceb70 0x7fa4bc1e3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc144980 0x7fa4bc0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1e4350 con 0x7fa4bc0b4370 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1e4630 con 0x7fa4bc144980 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4bc1e4910 con 0x7fa4bc0ceb70 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43046/0 (socket says 172.21.15.124:43046) 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 -- 172.21.15.124:0/1978928994 learned_addr learned my addr 172.21.15.124:0/1978928994 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4b9d9b640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc0ceb70 0x7fa4bc1e3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4bad9d640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc144980 0x7fa4bc0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 -- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc0ceb70 msgr2=0x7fa4bc1e3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc0ceb70 0x7fa4bc1e3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 -- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc144980 msgr2=0x7fa4bc0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc144980 0x7fa4bc0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 -- 172.21.15.124:0/1978928994 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ac009b30 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4ba59c640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00efc0 tx=0x7fa4ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac014480 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac00f040 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4b4009400 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac0147e0 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.595+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4bc1e4ed0 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.597+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4bc1e5090 con 0x7fa4bc0b4370 2023-12-13T23:56:15.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.597+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa4ac014940 con 0x7fa4bc0b4370 2023-12-13T23:56:15.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.597+0000 7fa4aaffd640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 0x7fa49c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.597+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa4ac089e10 con 0x7fa4bc0b4370 2023-12-13T23:56:15.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.597+0000 7fa4b9d9b640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 0x7fa49c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.598+0000 7fa4b9d9b640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 0x7fa49c06d680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc15a890 tx=0x7fa4b0007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.601+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa4ac054fd0 con 0x7fa4bc0b4370 2023-12-13T23:56:15.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2023-12-13T23:56:15.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.768+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4bc069400 con 0x7fa4bc0b4370 2023-12-13T23:56:15.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.769+0000 7fa4aaffd640 1 -- 172.21.15.124:0/1978928994 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12918 (secure 0 0 0) 0x7fa4ac0585f0 con 0x7fa4bc0b4370 2023-12-13T23:56:15.768 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:15.768 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":31,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:12.950196+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 msgr2=0x7fa49c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 0x7fa49c06d680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa4bc15a890 tx=0x7fa4b0007e80 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 msgr2=0x7fa4bc140ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00efc0 tx=0x7fa4ac00c490 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 shutdown_connections 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa49c06b1c0 0x7fa49c06d680 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa4bc144980 0x7fa4bc0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa4bc0ceb70 0x7fa4bc1e3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 --2- 172.21.15.124:0/1978928994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa4bc0b4370 0x7fa4bc140ab0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 >> 172.21.15.124:0/1978928994 conn(0x7fa4bc06a4d0 msgr2=0x7fa4bc157030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 shutdown_connections 2023-12-13T23:56:15.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.771+0000 7fa4c0cc9640 1 -- 172.21.15.124:0/1978928994 wait complete. 2023-12-13T23:56:15.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:15.781 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:15.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.885+0000 7f17ee96e640 1 Processor -- start 2023-12-13T23:56:15.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.885+0000 7f17ee96e640 1 -- start start 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17ee96e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 0x7f17e815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17ee96e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17e80ceb70 0x7f17e80cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17ee96e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17e8071770 con 0x7f17e80b4370 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17ee96e640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17e80ccdc0 con 0x7f17e80ceb70 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17ee96e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80cd0a0 con 0x7f17e8144980 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17e77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17e80ceb70 0x7f17e80cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17e77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f17e80ceb70 0x7f17e80cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43062/0 (socket says 172.21.15.124:43062) 2023-12-13T23:56:15.886 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.886+0000 7f17e77fe640 1 -- 172.21.15.124:0/63823644 learned_addr learned my addr 172.21.15.124:0/63823644 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e77fe640 1 -- 172.21.15.124:0/63823644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 msgr2=0x7f17e815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e7fff640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 0x7f17e815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e77fe640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 0x7f17e815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e77fe640 1 -- 172.21.15.124:0/63823644 >> v1:172.21.15.203:6789/0 conn(0x7f17e80b4370 legacy=0x7f17e8071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e77fe640 1 -- 172.21.15.124:0/63823644 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e80cd380 con 0x7f17e80ceb70 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e7fff640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 0x7f17e815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e77fe640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e80ceb70 0x7f17e80cc450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009870 tx=0x7f17dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4f1ea60993d41 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e6ffd640 1 -- 172.21.15.124:0/63823644 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc031640 con 0x7f17e80ceb70 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e6ffd640 1 -- 172.21.15.124:0/63823644 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17dc0317a0 con 0x7f17e80ceb70 2023-12-13T23:56:15.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.887+0000 7f17e6ffd640 1 -- 172.21.15.124:0/63823644 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17dc031b30 con 0x7f17e80ceb70 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 -- 172.21.15.124:0/63823644 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e80ceb70 msgr2=0x7f17e80cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e80ceb70 0x7f17e80cc450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009870 tx=0x7f17dc02b8f0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 -- 172.21.15.124:0/63823644 shutdown_connections 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e80ceb70 0x7f17e80cc450 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 --2- 172.21.15.124:0/63823644 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e8144980 0x7f17e815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 -- 172.21.15.124:0/63823644 >> 172.21.15.124:0/63823644 conn(0x7f17e806a4d0 msgr2=0x7f17e806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:15.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 -- 172.21.15.124:0/63823644 shutdown_connections 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.888+0000 7f17ee96e640 1 -- 172.21.15.124:0/63823644 wait complete. 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.889+0000 7f17ee96e640 1 Processor -- start 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.889+0000 7f17ee96e640 1 -- start start 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e80b4370 0x7f17e80aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e80ceb70 0x7f17e80b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80b24e0 con 0x7f17e8144980 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80b3060 con 0x7f17e80b4370 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ee96e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17e80b2730 con 0x7f17e80ceb70 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43068/0 (socket says 172.21.15.124:43068) 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 -- 172.21.15.124:0/2717451400 learned_addr learned my addr 172.21.15.124:0/2717451400 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:15.891 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17e7fff640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e80b4370 0x7f17e80aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17e77fe640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e80ceb70 0x7f17e80b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 -- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e80ceb70 msgr2=0x7f17e80b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e80ceb70 0x7f17e80b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 -- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e80b4370 msgr2=0x7f17e80aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e80b4370 0x7f17e80aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.890+0000 7f17ecf0b640 1 -- 172.21.15.124:0/2717451400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0094d0 con 0x7f17e8144980 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17ecf0b640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f17e000c780 tx=0x7f17e000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e0004b60 con 0x7f17e8144980 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17e000ce90 con 0x7f17e8144980 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e80b2a10 con 0x7f17e8144980 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17e000a850 con 0x7f17e8144980 2023-12-13T23:56:15.892 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.891+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e80b17a0 con 0x7f17e8144980 2023-12-13T23:56:15.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.892+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e80b1990 con 0x7f17e8144980 2023-12-13T23:56:15.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.892+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f17e000a9b0 con 0x7f17e8144980 2023-12-13T23:56:15.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.893+0000 7f17e4ff9640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 0x7f17d406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:15.893 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.893+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f17e00897e0 con 0x7f17e8144980 2023-12-13T23:56:15.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.893+0000 7f17e7fff640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 0x7f17d406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:15.894 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.893+0000 7f17e7fff640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 0x7f17d406d680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17d800a950 tx=0x7f17d8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:15.895 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:15.897+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f17e00548f0 con 0x7f17e8144980 2023-12-13T23:56:16.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.064+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f17e80b31d0 con 0x7f17e8144980 2023-12-13T23:56:16.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.064+0000 7f17e4ff9640 1 -- 172.21.15.124:0/2717451400 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f17e0057f10 con 0x7f17e8144980 2023-12-13T23:56:16.064 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 msgr2=0x7f17d406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 0x7f17d406d680 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f17d800a950 tx=0x7f17d8008040 comp rx=0 tx=0).stop 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 msgr2=0x7f17e80b34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f17e000c780 tx=0x7f17e000cc40 comp rx=0 tx=0).stop 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 shutdown_connections 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f17d406b1c0 0x7f17d406d680 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f17e8144980 0x7f17e80b34f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f17e80ceb70 0x7f17e80b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 --2- 172.21.15.124:0/2717451400 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f17e80b4370 0x7f17e80aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 >> 172.21.15.124:0/2717451400 conn(0x7f17e806a4d0 msgr2=0x7f17e81590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:16.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 shutdown_connections 2023-12-13T23:56:16.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.066+0000 7f17ee96e640 1 -- 172.21.15.124:0/2717451400 wait complete. 2023-12-13T23:56:16.076 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-13T23:56:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.176+0000 7f50f0fec640 1 Processor -- start 2023-12-13T23:56:16.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.176+0000 7f50f0fec640 1 -- start start 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 0x7f50ec0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50ec0d3730 0x7f50ec0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0b2b10 con 0x7f50ec0d3730 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50ec0d3b50 con 0x7f50ec15d330 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50f0fec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec0d1590 con 0x7f50ec0d2910 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.177+0000 7f50ead9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50ec0d3730 0x7f50ec0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50ec0d3730 0x7f50ec0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43072/0 (socket says 172.21.15.124:43072) 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 -- 172.21.15.124:0/3980333840 learned_addr learned my addr 172.21.15.124:0/3980333840 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:16.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ea59c640 1 --2- 172.21.15.124:0/3980333840 >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 0x7f50ec0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50e9d9b640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 -- 172.21.15.124:0/3980333840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 msgr2=0x7f50ec0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 -- 172.21.15.124:0/3980333840 >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 msgr2=0x7f50ec0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 --2- 172.21.15.124:0/3980333840 >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 0x7f50ec0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ead9d640 1 -- 172.21.15.124:0/3980333840 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ec0d1870 con 0x7f50ec0d3730 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50ea59c640 1 --2- 172.21.15.124:0/3980333840 >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 0x7f50ec0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.178+0000 7f50e9d9b640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50ead9d640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec0d3730 0x7f50ec0b6390 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0d0d10 tx=0x7f50e402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a78ff6251aaffd8 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50e959a640 1 -- 172.21.15.124:0/3980333840 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e40317e0 con 0x7f50ec0d3730 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50e959a640 1 -- 172.21.15.124:0/3980333840 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e4031940 con 0x7f50ec0d3730 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50e959a640 1 -- 172.21.15.124:0/3980333840 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4031cd0 con 0x7f50ec0d3730 2023-12-13T23:56:16.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 -- 172.21.15.124:0/3980333840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec0d3730 msgr2=0x7f50ec0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec0d3730 0x7f50ec0b6390 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f50ec0d0d10 tx=0x7f50e402b880 comp rx=0 tx=0).stop 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 -- 172.21.15.124:0/3980333840 shutdown_connections 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec0d3730 0x7f50ec0b6390 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 --2- 172.21.15.124:0/3980333840 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 --2- 172.21.15.124:0/3980333840 >> v2:172.21.15.203:3300/0 conn(0x7f50ec15d330 0x7f50ec0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 -- 172.21.15.124:0/3980333840 >> 172.21.15.124:0/3980333840 conn(0x7f50ec072020 msgr2=0x7f50ec072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 -- 172.21.15.124:0/3980333840 shutdown_connections 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.179+0000 7f50f0fec640 1 -- 172.21.15.124:0/3980333840 wait complete. 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 Processor -- start 2023-12-13T23:56:16.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 -- start start 2023-12-13T23:56:16.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0c9080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 0x7f50ec148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec15e890 con 0x7f50ec15d330 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec15ee10 con 0x7f50ec0d3730 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50f0fec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50ec1494e0 con 0x7f50ec0d2910 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50ead9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50ead9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43078/0 (socket says 172.21.15.124:43078) 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.181+0000 7f50ead9d640 1 -- 172.21.15.124:0/1180536122 learned_addr learned my addr 172.21.15.124:0/1180536122 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50e9d9b640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 0x7f50ec148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ea59c640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0c9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.185 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 -- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 msgr2=0x7f50ec0c9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0c9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 -- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 msgr2=0x7f50ec148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 0x7f50ec148fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 -- 172.21.15.124:0/1180536122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e40094d0 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ea59c640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50e9d9b640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 0x7f50ec148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50ead9d640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f50e40090b0 tx=0x7f50e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4032880 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e4032a20 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ec0c96d0 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4032dd0 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.182+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ec15eaa0 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.183+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50ec0d20d0 con 0x7f50ec15d330 2023-12-13T23:56:16.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.183+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f50e404e450 con 0x7f50ec15d330 2023-12-13T23:56:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.184+0000 7f50daffd640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.184+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f50e40ab770 con 0x7f50ec15d330 2023-12-13T23:56:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.184+0000 7f50ea59c640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.184+0000 7f50ea59c640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009540 tx=0x7f50dc00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.188+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f50e4076910 con 0x7f50ec15d330 2023-12-13T23:56:16.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.355+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f50ec0c6970 con 0x7f50ec15d330 2023-12-13T23:56:16.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.356+0000 7f50daffd640 1 -- 172.21.15.124:0/1180536122 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f50e4079f30 con 0x7f50ec15d330 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout:[ 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout: "poolnum": 1, 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout: "poolname": ".mgr" 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:16.355 INFO:teuthology.orchestra.run.smithi124.stdout:] 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 msgr2=0x7f50cc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f50dc009540 tx=0x7f50dc00b040 comp rx=0 tx=0).stop 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 msgr2=0x7f50ec1f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f50e40090b0 tx=0x7f50e40046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 shutdown_connections 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50ec0d3730 0x7f50ec148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50ec15d330 0x7f50ec1f6460 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.357+0000 7f50f0fec640 1 --2- 172.21.15.124:0/1180536122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50ec0d2910 0x7f50ec0c9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.358+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 >> 172.21.15.124:0/1180536122 conn(0x7f50ec072020 msgr2=0x7f50ec0b8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:16.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.358+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 shutdown_connections 2023-12-13T23:56:16.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.358+0000 7f50f0fec640 1 -- 172.21.15.124:0/1180536122 wait complete. 2023-12-13T23:56:16.367 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-13T23:56:16.367 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.472+0000 7f9274168640 1 Processor -- start 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 -- start start 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c143490 0x7f926c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f926c15abf0 0x7f926c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926c15e2a0 con 0x7f926c15e730 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f926c15b520 con 0x7f926c15abf0 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9274168640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c143870 con 0x7f926c143490 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.473+0000 7f9272705640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f926c15abf0 0x7f926c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9271703640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f926c15e730 0x7f926c15a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57852/0 (socket says 172.21.15.124:57852) 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9271703640 1 -- 172.21.15.124:0/61102395 learned_addr learned my addr 172.21.15.124:0/61102395 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:16.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9271f04640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c143490 0x7f926c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9272705640 1 -- 172.21.15.124:0/61102395 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c143490 msgr2=0x7f926c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:16.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9272705640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c143490 0x7f926c0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9272705640 1 -- 172.21.15.124:0/61102395 >> v1:172.21.15.124:6789/0 conn(0x7f926c15e730 legacy=0x7f926c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9272705640 1 -- 172.21.15.124:0/61102395 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c15b240 con 0x7f926c15abf0 2023-12-13T23:56:16.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.474+0000 7f9270f02640 1 -- 172.21.15.124:0/61102395 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553368422 0 0) 0x7f926c15e2a0 con 0x7f926c15e730 2023-12-13T23:56:16.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9272705640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c15abf0 0x7f926c0b1230 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f926400bfa0 tx=0x7f926402e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3871921aee98ca00 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9270f02640 1 -- 172.21.15.124:0/61102395 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926402ea40 con 0x7f926c15abf0 2023-12-13T23:56:16.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9270f02640 1 -- 172.21.15.124:0/61102395 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926402eba0 con 0x7f926c15abf0 2023-12-13T23:56:16.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9270f02640 1 -- 172.21.15.124:0/61102395 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264009540 con 0x7f926c15abf0 2023-12-13T23:56:16.477 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 -- 172.21.15.124:0/61102395 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c15abf0 msgr2=0x7f926c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c15abf0 0x7f926c0b1230 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f926400bfa0 tx=0x7f926402e0a0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 -- 172.21.15.124:0/61102395 shutdown_connections 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c15abf0 0x7f926c0b1230 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 --2- 172.21.15.124:0/61102395 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c143490 0x7f926c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 -- 172.21.15.124:0/61102395 >> 172.21.15.124:0/61102395 conn(0x7f926c06a4d0 msgr2=0x7f926c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 -- 172.21.15.124:0/61102395 shutdown_connections 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.475+0000 7f9274168640 1 -- 172.21.15.124:0/61102395 wait complete. 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.476+0000 7f9274168640 1 Processor -- start 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 -- start start 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 0x7f926c1599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f926c15e730 0x7f926c15cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c0c4660 con 0x7f926c143490 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f926c130fd0 con 0x7f926c15e730 2023-12-13T23:56:16.478 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9274168640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f926c131550 con 0x7f926c15abf0 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9271f04640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9271f04640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43094/0 (socket says 172.21.15.124:43094) 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.477+0000 7f9271f04640 1 -- 172.21.15.124:0/226039550 learned_addr learned my addr 172.21.15.124:0/226039550 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9272705640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f926c15e730 0x7f926c15cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271703640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 0x7f926c1599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 -- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 msgr2=0x7f926c1599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 0x7f926c1599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 -- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f926c15e730 msgr2=0x7f926c15cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f926c15e730 0x7f926c15cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 -- 172.21.15.124:0/226039550 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926400bc80 con 0x7f926c143490 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271703640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 0x7f926c1599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9271f04640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f926800d800 tx=0x7f926800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800b6c0 con 0x7f926c143490 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800b820 con 0x7f926c143490 2023-12-13T23:56:16.479 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800bbb0 con 0x7f926c143490 2023-12-13T23:56:16.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f926c133fb0 con 0x7f926c143490 2023-12-13T23:56:16.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.478+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f926c159f30 con 0x7f926c143490 2023-12-13T23:56:16.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.479+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f926c069400 con 0x7f926c143490 2023-12-13T23:56:16.480 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.479+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f926800bd10 con 0x7f926c143490 2023-12-13T23:56:16.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.480+0000 7f92627fc640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 0x7f925006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:16.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.480+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9268089bd0 con 0x7f926c143490 2023-12-13T23:56:16.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.480+0000 7f9271703640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 0x7f925006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:16.481 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.480+0000 7f9271703640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 0x7f925006d680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f926c0cebc0 tx=0x7f925c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:16.483 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.484+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9268054ce0 con 0x7f926c143490 2023-12-13T23:56:16.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.650+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f926c0b2820 con 0x7f926c143490 2023-12-13T23:56:16.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.969+0000 7f92627fc640 1 -- 172.21.15.124:0/226039550 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9268058300 con 0x7f926c143490 2023-12-13T23:56:16.968 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' created 2023-12-13T23:56:16.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 msgr2=0x7f925006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 0x7f925006d680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f926c0cebc0 tx=0x7f925c009040 comp rx=0 tx=0).stop 2023-12-13T23:56:16.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 msgr2=0x7f926c133a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:16.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f926800d800 tx=0x7f926800dcc0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 shutdown_connections 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f925006b1c0 0x7f925006d680 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f926c15e730 0x7f926c15cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f926c15abf0 0x7f926c1599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 --2- 172.21.15.124:0/226039550 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f926c143490 0x7f926c133a70 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 >> 172.21.15.124:0/226039550 conn(0x7f926c06a4d0 msgr2=0x7f926c0b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 shutdown_connections 2023-12-13T23:56:16.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:16.971+0000 7f9274168640 1 -- 172.21.15.124:0/226039550 wait complete. 2023-12-13T23:56:16.984 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-13T23:56:17.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 Processor -- start 2023-12-13T23:56:17.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 -- start start 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 0x7fc46815de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4680b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc468071d20 con 0x7fc468159920 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc468159d00 con 0x7fc468071010 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.093+0000 7fc46e44f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680b15c0 con 0x7fc4680b0bf0 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc46c9ec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4680b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc4677fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 0x7fc46815de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc467fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc468159920 0x7fc4680ce160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:42172/0 (socket says 172.21.15.124:42172) 2023-12-13T23:56:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc467fff640 1 -- 172.21.15.124:0/1554558035 learned_addr learned my addr 172.21.15.124:0/1554558035 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc466ffd640 1 -- 172.21.15.124:0/1554558035 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886263232 0 0) 0x7fc458001110 con 0x7fc468159920 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc466ffd640 1 -- 172.21.15.124:0/1554558035 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc450003520 con 0x7fc468159920 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc46c9ec640 1 -- 172.21.15.124:0/1554558035 >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 msgr2=0x7fc46815de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc46c9ec640 1 --2- 172.21.15.124:0/1554558035 >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 0x7fc46815de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc46c9ec640 1 -- 172.21.15.124:0/1554558035 >> v1:172.21.15.203:6789/0 conn(0x7fc468159920 legacy=0x7fc4680ce160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc46c9ec640 1 -- 172.21.15.124:0/1554558035 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680cdcd0 con 0x7fc4680b0bf0 2023-12-13T23:56:17.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.094+0000 7fc4677fe640 1 --2- 172.21.15.124:0/1554558035 >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 0x7fc46815de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.095+0000 7fc46c9ec640 1 --2- 172.21.15.124:0/1554558035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4680b1080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc460009d90 tx=0x7fc46002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6807d6a7951d75df server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.095+0000 7fc466ffd640 1 -- 172.21.15.124:0/1554558035 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc46002cbc0 con 0x7fc4680b0bf0 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.095+0000 7fc466ffd640 1 -- 172.21.15.124:0/1554558035 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc46002cd20 con 0x7fc4680b0bf0 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.095+0000 7fc466ffd640 1 -- 172.21.15.124:0/1554558035 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc460034640 con 0x7fc4680b0bf0 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 -- 172.21.15.124:0/1554558035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 msgr2=0x7fc4680b1080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1554558035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4680b1080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc460009d90 tx=0x7fc46002be10 comp rx=0 tx=0).stop 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 -- 172.21.15.124:0/1554558035 shutdown_connections 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1554558035 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4680b1080 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1554558035 >> v2:172.21.15.203:3300/0 conn(0x7fc468071010 0x7fc46815de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 -- 172.21.15.124:0/1554558035 >> 172.21.15.124:0/1554558035 conn(0x7fc46806d9c0 msgr2=0x7fc46806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 -- 172.21.15.124:0/1554558035 shutdown_connections 2023-12-13T23:56:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.096+0000 7fc46e44f640 1 -- 172.21.15.124:0/1554558035 wait complete. 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.097+0000 7fc46e44f640 1 Processor -- start 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.097+0000 7fc46e44f640 1 -- start start 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4681585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc468159920 0x7fc4680cb3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680a8570 con 0x7fc468071010 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680a8810 con 0x7fc468159920 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46e44f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4680a8af0 con 0x7fc4680b0bf0 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43096/0 (socket says 172.21.15.124:43096) 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 -- 172.21.15.124:0/1326748653 learned_addr learned my addr 172.21.15.124:0/1326748653 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc46c9ec640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc468159920 0x7fc4680cb3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 -- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 msgr2=0x7fc4681585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4681585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 -- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc468159920 msgr2=0x7fc4680cb3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc468159920 0x7fc4680cb3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 -- 172.21.15.124:0/1326748653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c009400 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.098+0000 7fc467fff640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c700 tx=0x7fc45800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458005ab0 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458005c10 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc460009a70 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458001d50 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc468158fc0 con 0x7fc468071010 2023-12-13T23:56:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.099+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4680a9010 con 0x7fc468071010 2023-12-13T23:56:17.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.100+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc458007070 con 0x7fc468071010 2023-12-13T23:56:17.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.100+0000 7fc464ff9640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 0x7fc45406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:17.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.101+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc458088cc0 con 0x7fc468071010 2023-12-13T23:56:17.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.101+0000 7fc4677fe640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 0x7fc45406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:17.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.101+0000 7fc4677fe640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 0x7fc45406d6b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006e80 tx=0x7fc45c006dd0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.105+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc4680a9010 con 0x7fc468071010 2023-12-13T23:56:17.272 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.273+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc468071d20 con 0x7fc468071010 2023-12-13T23:56:17.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.973+0000 7fc464ff9640 1 -- 172.21.15.124:0/1326748653 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc468071d20 con 0x7fc468071010 2023-12-13T23:56:17.972 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' created 2023-12-13T23:56:17.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.976+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 msgr2=0x7fc45406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.976+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 0x7fc45406d6b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc45c006e80 tx=0x7fc45c006dd0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.976+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 msgr2=0x7fc468151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:17.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.976+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c700 tx=0x7fc45800cbc0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 shutdown_connections 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc45406b1f0 0x7fc45406d6b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc468159920 0x7fc4680cb3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc4680b0bf0 0x7fc4681585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 --2- 172.21.15.124:0/1326748653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc468071010 0x7fc468151550 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:17.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.979+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 >> 172.21.15.124:0/1326748653 conn(0x7fc46806d9c0 msgr2=0x7fc4680703d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:17.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.981+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 shutdown_connections 2023-12-13T23:56:17.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:17.982+0000 7fc46e44f640 1 -- 172.21.15.124:0/1326748653 wait complete. 2023-12-13T23:56:17.999 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-13T23:56:18.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.103+0000 7f59f1e17640 1 Processor -- start 2023-12-13T23:56:18.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.103+0000 7f59f1e17640 1 -- start start 2023-12-13T23:56:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59f1e17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59ec15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59f1e17640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec071770 con 0x7f59ec0b4370 2023-12-13T23:56:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59f1e17640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ec0ccb60 con 0x7f59ec0ceb70 2023-12-13T23:56:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59f1e17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0ccdc0 con 0x7f59ec144980 2023-12-13T23:56:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59ebfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f59ec0b4370 0x7f59ec071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57864/0 (socket says 172.21.15.124:57864) 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59ebfff640 1 -- 172.21.15.124:0/2084778001 learned_addr learned my addr 172.21.15.124:0/2084778001 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59eb7fe640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59ec15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984402767 0 0) 0x7f59ec071770 con 0x7f59ec0b4370 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.104+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d4003520 con 0x7f59ec0b4370 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693506756 0 0) 0x7f59ec0ccb60 con 0x7f59ec0ceb70 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ec071770 con 0x7f59ec0ceb70 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543518860 0 0) 0x7f59d4003520 con 0x7f59ec0b4370 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 msgr2=0x7f59ec15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59ec15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 >> v1:172.21.15.203:6789/0 conn(0x7f59ec0ceb70 legacy=0x7f59ec0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0cd0a0 con 0x7f59ec0b4370 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f59e40019f0 con 0x7f59ec0b4370 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 >> v1:172.21.15.124:6789/0 conn(0x7f59ec0b4370 legacy=0x7f59ec071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59d4003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4004d00 0x7f59d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0ccdc0 con 0x7f59d40040e0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d400bac0 con 0x7f59d4004d00 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d400bc80 con 0x7f59ec144980 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eb7fe640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59d4003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59ebfff640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4004d00 0x7f59d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 -- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 msgr2=0x7f59d4003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec144980 0x7f59d4003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 -- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4004d00 msgr2=0x7f59d4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59d4004d00 0x7f59d4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.105+0000 7f59eaffd640 1 -- 172.21.15.124:0/2084778001 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d4003520 con 0x7f59d40040e0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59eaffd640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59d40047c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f59e00018a0 tx=0x7f59d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85ad04622d43befd server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e00052d0 con 0x7f59d40040e0 2023-12-13T23:56:18.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59e002e050 con 0x7f59d40040e0 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59ea7fc640 1 -- 172.21.15.124:0/2084778001 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59e0003810 con 0x7f59d40040e0 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59f1e17640 1 -- 172.21.15.124:0/2084778001 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 msgr2=0x7f59d40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59f1e17640 1 --2- 172.21.15.124:0/2084778001 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59d40047c0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f59e00018a0 tx=0x7f59d40057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.106+0000 7f59ebfff640 1 -- 172.21.15.124:0/2084778001 reap_dead start 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.107+0000 7f59f1e17640 1 -- 172.21.15.124:0/2084778001 shutdown_connections 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.107+0000 7f59f1e17640 1 -- 172.21.15.124:0/2084778001 >> 172.21.15.124:0/2084778001 conn(0x7f59ec06a4d0 msgr2=0x7f59ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.107+0000 7f59f1e17640 1 -- 172.21.15.124:0/2084778001 shutdown_connections 2023-12-13T23:56:18.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.107+0000 7f59f1e17640 1 -- 172.21.15.124:0/2084778001 wait complete. 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.108+0000 7f59f1e17640 1 Processor -- start 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.108+0000 7f59f1e17640 1 -- start start 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec0c9b90 0x7f59ec1550c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec0c9220 0x7f59ec15d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec13f2f0 con 0x7f59d40040e0 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec155600 con 0x7f59ec0c9220 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ec0b2290 con 0x7f59ec0c9b90 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43108/0 (socket says 172.21.15.124:43108) 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 -- 172.21.15.124:0/3949359364 learned_addr learned my addr 172.21.15.124:0/3949359364 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eaffd640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec0c9b90 0x7f59ec1550c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59ebfff640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec0c9220 0x7f59ec15d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 -- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec0c9b90 msgr2=0x7f59ec1550c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec0c9b90 0x7f59ec1550c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 -- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec0c9220 msgr2=0x7f59ec15d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec0c9220 0x7f59ec15d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 -- 172.21.15.124:0/3949359364 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ec0ca100 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59eb7fe640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0119a0 tx=0x7f59dc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc00faf0 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59dc00fc50 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ec0ca3e0 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59dc014440 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.109+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ec0ca6c0 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.110+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ec073c70 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.111+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f59dc0145a0 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.111+0000 7f59d3fff640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 0x7f59d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.111+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f59dc089f60 con 0x7f59d40040e0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.112+0000 7f59eaffd640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 0x7f59d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.112+0000 7f59eaffd640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 0x7f59d806d680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59e0033000 tx=0x7f59e0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:18.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.116+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f59dc054e30 con 0x7f59d40040e0 2023-12-13T23:56:18.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.319+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f59ec069400 con 0x7f59d40040e0 2023-12-13T23:56:18.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.984+0000 7f59d3fff640 1 -- 172.21.15.124:0/3949359364 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f59dc0220c0 con 0x7f59d40040e0 2023-12-13T23:56:18.983 INFO:teuthology.orchestra.run.smithi124.stderr:new fs with metadata pool 4 and data pool 5 2023-12-13T23:56:18.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.988+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 msgr2=0x7f59d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.988+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 0x7f59d806d680 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59e0033000 tx=0x7f59e0030000 comp rx=0 tx=0).stop 2023-12-13T23:56:18.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.988+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 msgr2=0x7f59ec13edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:18.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.988+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f59dc0119a0 tx=0x7f59dc011e60 comp rx=0 tx=0).stop 2023-12-13T23:56:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 shutdown_connections 2023-12-13T23:56:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f59d806b1c0 0x7f59d806d680 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f59e0033000 tx=0x7f59e0030000 comp rx=0 tx=0).stop 2023-12-13T23:56:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f59ec0c9220 0x7f59ec15d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f59ec0c9b90 0x7f59ec1550c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 --2- 172.21.15.124:0/3949359364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f59d40040e0 0x7f59ec13edb0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:18.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.989+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 >> 172.21.15.124:0/3949359364 conn(0x7f59ec06a4d0 msgr2=0x7f59ec143990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:18.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.993+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 shutdown_connections 2023-12-13T23:56:18.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:18.994+0000 7f59f1e17640 1 -- 172.21.15.124:0/3949359364 wait complete. 2023-12-13T23:56:19.020 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:19.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.121+0000 7fbb5449b640 1 Processor -- start 2023-12-13T23:56:19.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.121+0000 7fbb5449b640 1 -- start start 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb5449b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb5449b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb5449b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c071770 con 0x7fbb4c144980 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb5449b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4c15ced0 con 0x7fbb4c0ceb70 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb5449b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c0b4ce0 con 0x7fbb4c0b4370 2023-12-13T23:56:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb51a36640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb52a38640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb52a38640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:39296/0 (socket says 172.21.15.124:39296) 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb52a38640 1 -- 172.21.15.124:0/2659170023 learned_addr learned my addr 172.21.15.124:0/2659170023 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb52237640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fbb4c144980 0x7fbb4c15c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:42190/0 (socket says 172.21.15.124:42190) 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.122+0000 7fbb51a36640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:60064/0 (socket says 172.21.15.124:60064) 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb52a38640 1 -- 172.21.15.124:0/2659170023 >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 msgr2=0x7fbb4c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb52a38640 1 --2- 172.21.15.124:0/2659170023 >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb52a38640 1 -- 172.21.15.124:0/2659170023 >> v1:172.21.15.203:6789/0 conn(0x7fbb4c144980 legacy=0x7fbb4c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb52a38640 1 -- 172.21.15.124:0/2659170023 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0ccdc0 con 0x7fbb4c0b4370 2023-12-13T23:56:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb51a36640 1 --2- 172.21.15.124:0/2659170023 >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.123+0000 7fbb52a38640 1 --2- 172.21.15.124:0/2659170023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb4402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27291174d5ce54a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb51235640 1 -- 172.21.15.124:0/2659170023 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44030840 con 0x7fbb4c0b4370 2023-12-13T23:56:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb51235640 1 -- 172.21.15.124:0/2659170023 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb440309a0 con 0x7fbb4c0b4370 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb51235640 1 -- 172.21.15.124:0/2659170023 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44030d30 con 0x7fbb4c0b4370 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 -- 172.21.15.124:0/2659170023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 msgr2=0x7fbb4c0b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 --2- 172.21.15.124:0/2659170023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb4402b980 comp rx=0 tx=0).stop 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 -- 172.21.15.124:0/2659170023 shutdown_connections 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 --2- 172.21.15.124:0/2659170023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c0b47a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 --2- 172.21.15.124:0/2659170023 >> v2:172.21.15.203:3300/0 conn(0x7fbb4c0ceb70 0x7fbb4c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 -- 172.21.15.124:0/2659170023 >> 172.21.15.124:0/2659170023 conn(0x7fbb4c06a4d0 msgr2=0x7fbb4c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 -- 172.21.15.124:0/2659170023 shutdown_connections 2023-12-13T23:56:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.124+0000 7fbb5449b640 1 -- 172.21.15.124:0/2659170023 wait complete. 2023-12-13T23:56:19.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.125+0000 7fbb5449b640 1 Processor -- start 2023-12-13T23:56:19.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.125+0000 7fbb5449b640 1 -- start start 2023-12-13T23:56:19.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb4c0ceb70 0x7fbb4c0c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c0c51a0 con 0x7fbb4c144980 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c157720 con 0x7fbb4c0ceb70 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb5449b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c157a00 con 0x7fbb4c0b4370 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43114/0 (socket says 172.21.15.124:43114) 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 -- 172.21.15.124:0/501638934 learned_addr learned my addr 172.21.15.124:0/501638934 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52237640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb51a36640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb4c0ceb70 0x7fbb4c0c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 -- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 msgr2=0x7fbb4c154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 -- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb4c0ceb70 msgr2=0x7fbb4c0c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb4c0ceb70 0x7fbb4c0c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.126+0000 7fbb52a38640 1 -- 172.21.15.124:0/501638934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb40009400 con 0x7fbb4c144980 2023-12-13T23:56:19.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb52a38640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44039a90 con 0x7fbb4c144980 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb44039c10 con 0x7fbb4c144980 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb440094d0 con 0x7fbb4c144980 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44043410 con 0x7fbb4c144980 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.127+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c157fc0 con 0x7fbb4c144980 2023-12-13T23:56:19.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.128+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4c069400 con 0x7fbb4c144980 2023-12-13T23:56:19.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.128+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbb4402f390 con 0x7fbb4c144980 2023-12-13T23:56:19.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.129+0000 7fbb4affd640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 0x7fbb3806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.129+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb440aba50 con 0x7fbb4c144980 2023-12-13T23:56:19.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.129+0000 7fbb52237640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 0x7fbb3806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.129+0000 7fbb52237640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 0x7fbb3806d680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0012e0 tx=0x7fbb3c0016f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.131 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.133+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbb44076840 con 0x7fbb4c144980 2023-12-13T23:56:19.302 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.303+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb4c0c53d0 con 0x7fbb4c144980 2023-12-13T23:56:19.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.305+0000 7fbb4affd640 1 -- 172.21.15.124:0/501638934 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16643 (secure 0 0 0) 0x7fbb44079e60 con 0x7fbb4c144980 2023-12-13T23:56:19.304 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:19.304 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":34,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:18.320417+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 msgr2=0x7fbb3806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 0x7fbb3806d680 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0012e0 tx=0x7fbb3c0016f0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 msgr2=0x7fbb4c0c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbb44009870 tx=0x7fbb44004730 comp rx=0 tx=0).stop 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 shutdown_connections 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbb3806b1c0 0x7fbb3806d680 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbb4c0ceb70 0x7fbb4c0c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb4c144980 0x7fbb4c0c7650 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.306 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 --2- 172.21.15.124:0/501638934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbb4c0b4370 0x7fbb4c154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 >> 172.21.15.124:0/501638934 conn(0x7fbb4c06a4d0 msgr2=0x7fbb4c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:19.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 shutdown_connections 2023-12-13T23:56:19.307 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.307+0000 7fbb5449b640 1 -- 172.21.15.124:0/501638934 wait complete. 2023-12-13T23:56:19.317 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:19.416 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.418+0000 7f7600325640 1 Processor -- start 2023-12-13T23:56:19.417 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.418+0000 7f7600325640 1 -- start start 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f7600325640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f75f8150940 0x7f75f815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f7600325640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f80b49f0 con 0x7f75f815dec0 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f7600325640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f80b3560 con 0x7f75f8150560 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f7600325640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f80b3de0 con 0x7f75f8150940 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fe8c2640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f75f815dec0 0x7f75f80ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57880/0 (socket says 172.21.15.124:57880) 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fe8c2640 1 -- 172.21.15.124:0/1047145705 learned_addr learned my addr 172.21.15.124:0/1047145705 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:19.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd8c0640 1 --2- 172.21.15.124:0/1047145705 >> v2:172.21.15.203:3300/0 conn(0x7f75f8150940 0x7f75f815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440385483 0 0) 0x7f75f80b49f0 con 0x7f75f815dec0 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e4003520 con 0x7f75f815dec0 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd8c0640 1 -- 172.21.15.124:0/1047145705 >> v1:172.21.15.203:6789/0 conn(0x7f75f8150560 legacy=0x7f75f80b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504065044 0 0) 0x7f75f80b3560 con 0x7f75f8150560 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd8c0640 1 -- 172.21.15.124:0/1047145705 >> v1:172.21.15.124:6789/0 conn(0x7f75f815dec0 legacy=0x7f75f80ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.419+0000 7f75fd8c0640 1 -- 172.21.15.124:0/1047145705 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80b40c0 con 0x7f75f8150940 2023-12-13T23:56:19.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.420+0000 7f75fd8c0640 1 --2- 172.21.15.124:0/1047145705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8150940 0x7f75f815d200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f75e800a940 tx=0x7f75e802cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f48ccfa3bef8be64 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.420+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f00019f0 con 0x7f75f8150940 2023-12-13T23:56:19.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.420+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e802d750 con 0x7f75f8150940 2023-12-13T23:56:19.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.420+0000 7f75fd0bf640 1 -- 172.21.15.124:0/1047145705 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e802dae0 con 0x7f75f8150940 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 -- 172.21.15.124:0/1047145705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8150940 msgr2=0x7f75f815d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 --2- 172.21.15.124:0/1047145705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8150940 0x7f75f815d200 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f75e800a940 tx=0x7f75e802cc20 comp rx=0 tx=0).stop 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 -- 172.21.15.124:0/1047145705 shutdown_connections 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 --2- 172.21.15.124:0/1047145705 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f8150940 0x7f75f815d200 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 -- 172.21.15.124:0/1047145705 >> 172.21.15.124:0/1047145705 conn(0x7f75f806a4d0 msgr2=0x7f75f806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 -- 172.21.15.124:0/1047145705 shutdown_connections 2023-12-13T23:56:19.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.421+0000 7f7600325640 1 -- 172.21.15.124:0/1047145705 wait complete. 2023-12-13T23:56:19.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.422+0000 7f7600325640 1 Processor -- start 2023-12-13T23:56:19.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.422+0000 7f7600325640 1 -- start start 2023-12-13T23:56:19.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f815dec0 0x7f75f814b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f75f814ba90 0x7f75f80af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f814cc20 con 0x7f75f8150560 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f814c010 con 0x7f75f815dec0 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f7600325640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f814c2f0 con 0x7f75f814ba90 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43120/0 (socket says 172.21.15.124:43120) 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 -- 172.21.15.124:0/1073766904 learned_addr learned my addr 172.21.15.124:0/1073766904 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 -- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f75f814ba90 msgr2=0x7f75f80af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f75f814ba90 0x7f75f80af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 -- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f815dec0 msgr2=0x7f75f814b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f815dec0 0x7f75f814b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 -- 172.21.15.124:0/1073766904 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e800a5a0 con 0x7f75f8150560 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.423+0000 7f75fe0c1640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75f400c410 tx=0x7f75f400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.424+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40035f0 con 0x7f75f8150560 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.424+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4003770 con 0x7f75f8150560 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.424+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f814c5d0 con 0x7f75f8150560 2023-12-13T23:56:19.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.424+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4003b20 con 0x7f75f8150560 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.424+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f814b100 con 0x7f75f8150560 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.425+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f8069400 con 0x7f75f8150560 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.425+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f75f4003c80 con 0x7f75f8150560 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.425+0000 7f75eeffd640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 0x7f75dc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.425+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75f4089050 con 0x7f75f8150560 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.426+0000 7f75fd8c0640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 0x7f75dc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.426+0000 7f75fd8c0640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 0x7f75dc06d680 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f75e8037000 tx=0x7f75e802f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.429+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f75f4053d80 con 0x7f75f8150560 2023-12-13T23:56:19.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.596+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75f815cdc0 con 0x7f75f8150560 2023-12-13T23:56:19.596 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.597+0000 7f75eeffd640 1 -- 172.21.15.124:0/1073766904 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16643 (secure 0 0 0) 0x7f75f4017090 con 0x7f75f8150560 2023-12-13T23:56:19.597 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:19.597 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":34,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:18.320417+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:19.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 msgr2=0x7f75dc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 0x7f75dc06d680 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f75e8037000 tx=0x7f75e802f000 comp rx=0 tx=0).stop 2023-12-13T23:56:19.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 msgr2=0x7f75f814d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f75f400c410 tx=0x7f75f400c8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 shutdown_connections 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f75dc06b1c0 0x7f75dc06d680 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f75f815dec0 0x7f75f814b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f75f814ba90 0x7f75f80af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 --2- 172.21.15.124:0/1073766904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f75f8150560 0x7f75f814d0b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 >> 172.21.15.124:0/1073766904 conn(0x7f75f806a4d0 msgr2=0x7f75f80cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 shutdown_connections 2023-12-13T23:56:19.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.599+0000 7f7600325640 1 -- 172.21.15.124:0/1073766904 wait complete. 2023-12-13T23:56:19.610 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-13T23:56:19.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.714+0000 7fce54eec640 1 Processor -- start 2023-12-13T23:56:19.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.714+0000 7fce54eec640 1 -- start start 2023-12-13T23:56:19.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fce5015d1d0 0x7fce5006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fce500ad640 0x7fce5006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce500aeb80 0x7fce50160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce500adae0 con 0x7fce500ad640 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce500addc0 con 0x7fce5015d1d0 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce54eec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500ae0a0 con 0x7fce500aeb80 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fce500ad640 0x7fce5006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fce500ad640 0x7fce5006ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43122/0 (socket says 172.21.15.124:43122) 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 -- 172.21.15.124:0/1880010232 learned_addr learned my addr 172.21.15.124:0/1880010232 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4e59c640 1 --2- 172.21.15.124:0/1880010232 >> v2:172.21.15.203:3300/0 conn(0x7fce5015d1d0 0x7fce5006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 -- 172.21.15.124:0/1880010232 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce500aeb80 msgr2=0x7fce50160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4ed9d640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce500aeb80 0x7fce50160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce500aeb80 0x7fce50160ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.715 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 -- 172.21.15.124:0/1880010232 >> v2:172.21.15.203:3300/0 conn(0x7fce5015d1d0 msgr2=0x7fce5006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/1880010232 >> v2:172.21.15.203:3300/0 conn(0x7fce5015d1d0 0x7fce5006b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.715+0000 7fce4dd9b640 1 -- 172.21.15.124:0/1880010232 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce500ae380 con 0x7fce500ad640 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.716+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500ad640 0x7fce5006ba20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fce50072310 tx=0x7fce4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7bb626d5fab438 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.716+0000 7fce4d59a640 1 -- 172.21.15.124:0/1880010232 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44031870 con 0x7fce500ad640 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.716+0000 7fce4d59a640 1 -- 172.21.15.124:0/1880010232 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce440319d0 con 0x7fce500ad640 2023-12-13T23:56:19.716 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.716+0000 7fce4d59a640 1 -- 172.21.15.124:0/1880010232 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44031d60 con 0x7fce500ad640 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 -- 172.21.15.124:0/1880010232 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500ad640 msgr2=0x7fce5006ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500ad640 0x7fce5006ba20 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fce50072310 tx=0x7fce4402b910 comp rx=0 tx=0).stop 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 -- 172.21.15.124:0/1880010232 shutdown_connections 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500ad640 0x7fce5006ba20 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 --2- 172.21.15.124:0/1880010232 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce500aeb80 0x7fce50160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 --2- 172.21.15.124:0/1880010232 >> v2:172.21.15.203:3300/0 conn(0x7fce5015d1d0 0x7fce5006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 -- 172.21.15.124:0/1880010232 >> 172.21.15.124:0/1880010232 conn(0x7fce50070e70 msgr2=0x7fce50071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:19.717 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 -- 172.21.15.124:0/1880010232 shutdown_connections 2023-12-13T23:56:19.718 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.717+0000 7fce54eec640 1 -- 172.21.15.124:0/1880010232 wait complete. 2023-12-13T23:56:19.718 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.718+0000 7fce54eec640 1 Processor -- start 2023-12-13T23:56:19.718 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.718+0000 7fce54eec640 1 -- start start 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce500ad640 0x7fce5014b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 0x7fce50155030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015e9c0 con 0x7fce500aeb80 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5014b940 con 0x7fce500ad640 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce54eec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015ddb0 con 0x7fce5015d1d0 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.720 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43138/0 (socket says 172.21.15.124:43138) 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 -- 172.21.15.124:0/4037013021 learned_addr learned my addr 172.21.15.124:0/4037013021 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4e59c640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce500ad640 0x7fce5014b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4ed9d640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 0x7fce50155030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 -- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 msgr2=0x7fce50155030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 0x7fce50155030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 -- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce500ad640 msgr2=0x7fce5014b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce500ad640 0x7fce5014b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 -- 172.21.15.124:0/4037013021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce440094d0 con 0x7fce500aeb80 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4ed9d640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 0x7fce50155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.719+0000 7fce4dd9b640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce44000c00 tx=0x7fce440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.720+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce44031860 con 0x7fce500aeb80 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.720+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce440443f0 con 0x7fce500aeb80 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.720+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce440447a0 con 0x7fce500aeb80 2023-12-13T23:56:19.721 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.720+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce5015e090 con 0x7fce500aeb80 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.720+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce5015e650 con 0x7fce500aeb80 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.721+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5015eb70 con 0x7fce500aeb80 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.721+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fce44044900 con 0x7fce500aeb80 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.721+0000 7fce3effd640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 0x7fce3006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.722+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce4402d040 con 0x7fce500aeb80 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.722+0000 7fce4e59c640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 0x7fce3006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:19.722 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.722+0000 7fce4e59c640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 0x7fce3006d680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce4000f360 tx=0x7fce40007710 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:19.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.726+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fce44077190 con 0x7fce500aeb80 2023-12-13T23:56:19.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:19.925+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fce50069400 con 0x7fce500aeb80 2023-12-13T23:56:20.004 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.005+0000 7fce3effd640 1 -- 172.21.15.124:0/4037013021 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v13) v1 ==== 118+0+0 (secure 0 0 0) 0x7fce4407a7b0 con 0x7fce500aeb80 2023-12-13T23:56:20.006 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 msgr2=0x7fce3006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.006 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 0x7fce3006d680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fce4000f360 tx=0x7fce40007710 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 msgr2=0x7fce5015a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fce44000c00 tx=0x7fce440046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 shutdown_connections 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fce3006b1c0 0x7fce3006d680 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fce5015d1d0 0x7fce50155030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.007+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fce500aeb80 0x7fce5015a130 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.008+0000 7fce54eec640 1 --2- 172.21.15.124:0/4037013021 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fce500ad640 0x7fce5014b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.008+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 >> 172.21.15.124:0/4037013021 conn(0x7fce50070e70 msgr2=0x7fce5006b360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.008+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 shutdown_connections 2023-12-13T23:56:20.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.008+0000 7fce54eec640 1 -- 172.21.15.124:0/4037013021 wait complete. 2023-12-13T23:56:20.017 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-13T23:56:20.017 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-13T23:56:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.121+0000 7fa80bccf640 1 Processor -- start 2023-12-13T23:56:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 -- start start 2023-12-13T23:56:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa80415d330 0x7fa8040b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa8040d2910 0x7fa8040d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8040b2b10 con 0x7fa8040d3730 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8040d3bc0 con 0x7fa80415d330 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.122+0000 7fa80bccf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8040d1590 con 0x7fa8040d2910 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa809a6b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa80415d330 0x7fa8040b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa80a26c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa8040d3730 0x7fa8040b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:57882/0 (socket says 172.21.15.124:57882) 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa80a26c640 1 -- 172.21.15.124:0/1984748241 learned_addr learned my addr 172.21.15.124:0/1984748241 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa80926a640 1 --2- 172.21.15.124:0/1984748241 >> v2:172.21.15.203:3300/0 conn(0x7fa8040d2910 0x7fa8040d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa809a6b640 1 -- 172.21.15.124:0/1984748241 >> v2:172.21.15.203:3300/0 conn(0x7fa8040d2910 msgr2=0x7fa8040d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa809a6b640 1 --2- 172.21.15.124:0/1984748241 >> v2:172.21.15.203:3300/0 conn(0x7fa8040d2910 0x7fa8040d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa809a6b640 1 -- 172.21.15.124:0/1984748241 >> v1:172.21.15.124:6789/0 conn(0x7fa8040d3730 legacy=0x7fa8040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.123+0000 7fa809a6b640 1 -- 172.21.15.124:0/1984748241 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8040d1870 con 0x7fa80415d330 2023-12-13T23:56:20.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa809a6b640 1 --2- 172.21.15.124:0/1984748241 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa80415d330 0x7fa8040b8b80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa8040d1540 tx=0x7fa7fc02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e90cff8684877191 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa808a69640 1 -- 172.21.15.124:0/1984748241 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc02cbf0 con 0x7fa80415d330 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa808a69640 1 -- 172.21.15.124:0/1984748241 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7fc02cd50 con 0x7fa80415d330 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa808a69640 1 -- 172.21.15.124:0/1984748241 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc034660 con 0x7fa80415d330 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa80bccf640 1 -- 172.21.15.124:0/1984748241 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa80415d330 msgr2=0x7fa8040b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.124+0000 7fa80bccf640 1 --2- 172.21.15.124:0/1984748241 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa80415d330 0x7fa8040b8b80 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa8040d1540 tx=0x7fa7fc02be40 comp rx=0 tx=0).stop 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 -- 172.21.15.124:0/1984748241 shutdown_connections 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 --2- 172.21.15.124:0/1984748241 >> v2:172.21.15.203:3300/0 conn(0x7fa8040d2910 0x7fa8040d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 --2- 172.21.15.124:0/1984748241 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa80415d330 0x7fa8040b8b80 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 -- 172.21.15.124:0/1984748241 >> 172.21.15.124:0/1984748241 conn(0x7fa804072020 msgr2=0x7fa804072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 -- 172.21.15.124:0/1984748241 shutdown_connections 2023-12-13T23:56:20.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.125+0000 7fa80bccf640 1 -- 172.21.15.124:0/1984748241 wait complete. 2023-12-13T23:56:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.126+0000 7fa80bccf640 1 Processor -- start 2023-12-13T23:56:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.126+0000 7fa80bccf640 1 -- start start 2023-12-13T23:56:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8040d2910 0x7fa80415e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 0x7fa8041f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415eb20 con 0x7fa8040d3730 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415f630 con 0x7fa8040d2910 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80415f910 con 0x7fa80415d330 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:43156/0 (socket says 172.21.15.124:43156) 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 -- 172.21.15.124:0/3924726517 learned_addr learned my addr 172.21.15.124:0/3924726517 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa809a6b640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8040d2910 0x7fa80415e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80a26c640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 0x7fa8041f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 -- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 msgr2=0x7fa8041f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 0x7fa8041f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 -- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8040d2910 msgr2=0x7fa80415e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8040d2910 0x7fa80415e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 -- 172.21.15.124:0/3924726517 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7fc009a70 con 0x7fa8040d3730 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80a26c640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 0x7fa8041f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80926a640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e9b0 tx=0x7fa7f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4009670 con 0x7fa8040d3730 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f400c9a0 con 0x7fa8040d3730 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80415fbf0 con 0x7fa8040d3730 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f400cd50 con 0x7fa8040d3730 2023-12-13T23:56:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.127+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8041601b0 con 0x7fa8040d3730 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.128+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa804069400 con 0x7fa8040d3730 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.129+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa7f4013470 con 0x7fa8040d3730 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.129+0000 7fa7fa7fc640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 0x7fa7e806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.129+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7f4089b30 con 0x7fa8040d3730 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.129+0000 7fa809a6b640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 0x7fa7e806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.130+0000 7fa809a6b640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 0x7fa7e806d680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc036000 tx=0x7fa7fc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:20.132 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.133+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa7f40549a0 con 0x7fa8040d3730 2023-12-13T23:56:20.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:20.336+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa8040b65a0 con 0x7fa8040d3730 2023-12-13T23:56:21.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.011+0000 7fa7fa7fc640 1 -- 172.21.15.124:0/3924726517 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa7f40170c0 con 0x7fa8040d3730 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 msgr2=0x7fa7e806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 0x7fa7e806d680 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc036000 tx=0x7fa7fc030000 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 msgr2=0x7fa8041f31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fa7f400e9b0 tx=0x7fa7f400ee70 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 shutdown_connections 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa7e806b1c0 0x7fa7e806d680 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa80415d330 0x7fa8041f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa8040d3730 0x7fa8041f31f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 --2- 172.21.15.124:0/3924726517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa8040d2910 0x7fa80415e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 >> 172.21.15.124:0/3924726517 conn(0x7fa804072020 msgr2=0x7fa8040b6a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 shutdown_connections 2023-12-13T23:56:21.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.013+0000 7fa80bccf640 1 -- 172.21.15.124:0/3924726517 wait complete. 2023-12-13T23:56:21.024 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-13T23:56:21.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:56:21.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.126+0000 7f35fcf78640 1 Processor -- start 2023-12-13T23:56:21.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.126+0000 7f35fcf78640 1 -- start start 2023-12-13T23:56:21.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35fcf78640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f35f815d330 0x7f35f80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35fcf78640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35f80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35fcf78640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f80b2b10 con 0x7f35f80d2910 2023-12-13T23:56:21.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35fcf78640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80d1590 con 0x7f35f815d330 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35fcf78640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80d1870 con 0x7f35f80d3900 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35f5d9b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f35f80d2910 0x7f35f80d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36182/0 (socket says 172.21.15.124:36182) 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.127+0000 7f35f5d9b640 1 -- 172.21.15.124:0/3535181420 learned_addr learned my addr 172.21.15.124:0/3535181420 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35f80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f659c640 1 --2- 172.21.15.124:0/3535181420 >> v2:172.21.15.203:3300/0 conn(0x7f35f815d330 0x7f35f80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917362396 0 0) 0x7f35f80b2b10 con 0x7f35f80d2910 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e0003520 con 0x7f35f80d2910 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3078997413 0 0) 0x7f35e0003520 con 0x7f35f80d2910 2023-12-13T23:56:21.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 msgr2=0x7f35f80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35f80b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 >> v2:172.21.15.203:3300/0 conn(0x7f35f815d330 msgr2=0x7f35f80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 --2- 172.21.15.124:0/3535181420 >> v2:172.21.15.203:3300/0 conn(0x7f35f815d330 0x7f35f80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80d1b50 con 0x7f35f80d2910 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f35ec001670 con 0x7f35f80d2910 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 >> v1:172.21.15.124:6789/0 conn(0x7f35f80d2910 legacy=0x7f35f80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35e0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35e0003ef0 0x7f35e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35e0004d00 0x7f35e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80d1590 con 0x7f35e0004d00 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80d1870 con 0x7f35e0003ef0 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e000b9d0 con 0x7f35f80d3900 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35e0004d00 0x7f35e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f659c640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35e0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f5d9b640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35e0003ef0 0x7f35e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 -- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 msgr2=0x7f35e0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35f80d3900 0x7f35e0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 -- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35e0003ef0 msgr2=0x7f35e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35e0003ef0 0x7f35e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.128+0000 7f35f6d9d640 1 -- 172.21.15.124:0/3535181420 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e0003520 con 0x7f35e0004d00 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.129+0000 7f35f6d9d640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35e0004d00 0x7f35e0005180 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009af0 tx=0x7f35e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fef9deef9153190e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.129+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f002c6b0 con 0x7f35e0004d00 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.129+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35f002c810 con 0x7f35e0004d00 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.129+0000 7f35f559a640 1 -- 172.21.15.124:0/3535181420 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f002cba0 con 0x7f35e0004d00 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 -- 172.21.15.124:0/3535181420 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35e0004d00 msgr2=0x7f35e0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3535181420 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35e0004d00 0x7f35e0005180 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f35f0009af0 tx=0x7f35e00057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35f6d9d640 1 -- 172.21.15.124:0/3535181420 reap_dead start 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 -- 172.21.15.124:0/3535181420 shutdown_connections 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 -- 172.21.15.124:0/3535181420 >> 172.21.15.124:0/3535181420 conn(0x7f35f8072020 msgr2=0x7f35f8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 -- 172.21.15.124:0/3535181420 shutdown_connections 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.130+0000 7f35fcf78640 1 -- 172.21.15.124:0/3535181420 wait complete. 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.131+0000 7f35fcf78640 1 Processor -- start 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.131+0000 7f35fcf78640 1 -- start start 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35e0004d00 0x7f35f8153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f81557a0 0x7f35f80c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c7900 con 0x7f35f8154de0 2023-12-13T23:56:21.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f81539c0 con 0x7f35f81557a0 2023-12-13T23:56:21.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35fcf78640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80c7280 con 0x7f35e0004d00 2023-12-13T23:56:21.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49350/0 (socket says 172.21.15.124:49350) 2023-12-13T23:56:21.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 -- 172.21.15.124:0/3313385279 learned_addr learned my addr 172.21.15.124:0/3313385279 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 -- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35e0004d00 msgr2=0x7f35f8153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35e0004d00 0x7f35f8153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 -- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f81557a0 msgr2=0x7f35f80c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f81557a0 0x7f35f80c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.132+0000 7f35f5d9b640 1 -- 172.21.15.124:0/3313385279 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f80cd090 con 0x7f35f8154de0 2023-12-13T23:56:21.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35f5d9b640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f35ec000fb0 tx=0x7f35ec0077a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec00f940 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35ec00faa0 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35ec00fe30 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f80cd370 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.133+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f8155c20 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.134+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f80c7a40 con 0x7f35f8154de0 2023-12-13T23:56:21.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.134+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f35ec01d460 con 0x7f35f8154de0 2023-12-13T23:56:21.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.134+0000 7f35e6ffd640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 0x7f35d806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.135+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35ec089bf0 con 0x7f35f8154de0 2023-12-13T23:56:21.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.135+0000 7f35f659c640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 0x7f35d806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.135+0000 7f35f659c640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 0x7f35d806d6b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009fc0 tx=0x7f35e8009750 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.139+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f35f80c7a40 con 0x7f35f8154de0 2023-12-13T23:56:21.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.339+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35f80c7a40 con 0x7f35f8154de0 2023-12-13T23:56:21.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.339+0000 7f35e6ffd640 1 -- 172.21.15.124:0/3313385279 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f35f80c7a40 con 0x7f35f8154de0 2023-12-13T23:56:21.338 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:21.338 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:20.337597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:56:21.339 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 14 2023-12-13T23:56:21.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 msgr2=0x7f35d806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 0x7f35d806d6b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009fc0 tx=0x7f35e8009750 comp rx=0 tx=0).stop 2023-12-13T23:56:21.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 msgr2=0x7f35f8155260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f35ec000fb0 tx=0x7f35ec0077a0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 shutdown_connections 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f35d806b1f0 0x7f35d806d6b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f35f81557a0 0x7f35f80c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f35f8154de0 0x7f35f8155260 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 --2- 172.21.15.124:0/3313385279 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f35e0004d00 0x7f35f8153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 >> 172.21.15.124:0/3313385279 conn(0x7f35f8072020 msgr2=0x7f35f80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 shutdown_connections 2023-12-13T23:56:21.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.341+0000 7f35fcf78640 1 -- 172.21.15.124:0/3313385279 wait complete. 2023-12-13T23:56:21.351 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:21.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.452+0000 7f6a08bfb640 1 Processor -- start 2023-12-13T23:56:21.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.452+0000 7f6a08bfb640 1 -- start start 2023-12-13T23:56:21.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a08bfb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6a0415d330 0x7f6a040b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a08bfb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f6a040b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a08bfb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a040b2b10 con 0x7f6a040d2910 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a08bfb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a040d1590 con 0x7f6a0415d330 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a08bfb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040d1870 con 0x7f6a040d3900 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a01d9b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6a040d2910 0x7f6a040d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36196/0 (socket says 172.21.15.124:36196) 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.453+0000 7f6a01d9b640 1 -- 172.21.15.124:0/995654009 learned_addr learned my addr 172.21.15.124:0/995654009 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364681925 0 0) 0x7f6a040b2b10 con 0x7f6a040d2910 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a02d9d640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f6a040b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0259c640 1 --2- 172.21.15.124:0/995654009 >> v2:172.21.15.203:3300/0 conn(0x7f6a0415d330 0x7f6a040b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec003520 con 0x7f6a040d2910 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266896067 0 0) 0x7f69ec003520 con 0x7f6a040d2910 2023-12-13T23:56:21.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 msgr2=0x7f6a040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f6a040b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 >> v2:172.21.15.203:3300/0 conn(0x7f6a0415d330 msgr2=0x7f6a040b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 --2- 172.21.15.124:0/995654009 >> v2:172.21.15.203:3300/0 conn(0x7f6a0415d330 0x7f6a040b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040d1b50 con 0x7f6a040d2910 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f69f8001670 con 0x7f6a040d2910 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 >> v1:172.21.15.124:6789/0 conn(0x7f6a040d2910 legacy=0x7f6a040d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec003970 0x7f69ec003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69ec0047c0 0x7f69ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f69ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040d1590 con 0x7f69ec0047c0 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040d1870 con 0x7f69ec003970 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec00b980 con 0x7f6a040d3900 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69ec0047c0 0x7f69ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a0259c640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec003970 0x7f69ec003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a02d9d640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f69ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 -- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 msgr2=0x7f69ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a040d3900 0x7f69ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 -- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec003970 msgr2=0x7f69ec003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec003970 0x7f69ec003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.454+0000 7f6a01d9b640 1 -- 172.21.15.124:0/995654009 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec003520 con 0x7f69ec0047c0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.455+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69ec0047c0 0x7f69ec004c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f69f800c270 tx=0x7f69ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a777644480fa74a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.455+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f802cae0 con 0x7f69ec0047c0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.455+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f802cc40 con 0x7f69ec0047c0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.455+0000 7f6a0159a640 1 -- 172.21.15.124:0/995654009 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f80058d0 con 0x7f69ec0047c0 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 -- 172.21.15.124:0/995654009 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69ec0047c0 msgr2=0x7f69ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/995654009 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69ec0047c0 0x7f69ec004c40 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f69f800c270 tx=0x7f69ec0057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a02d9d640 1 -- 172.21.15.124:0/995654009 reap_dead start 2023-12-13T23:56:21.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 -- 172.21.15.124:0/995654009 shutdown_connections 2023-12-13T23:56:21.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 -- 172.21.15.124:0/995654009 >> 172.21.15.124:0/995654009 conn(0x7f6a04072020 msgr2=0x7f6a04072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 -- 172.21.15.124:0/995654009 shutdown_connections 2023-12-13T23:56:21.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.456+0000 7f6a08bfb640 1 -- 172.21.15.124:0/995654009 wait complete. 2023-12-13T23:56:21.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.457+0000 7f6a08bfb640 1 Processor -- start 2023-12-13T23:56:21.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.457+0000 7f6a08bfb640 1 -- start start 2023-12-13T23:56:21.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 0x7f6a040cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 0x7f6a04159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cd530 con 0x7f6a041f1b40 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cc670 con 0x7f69ec0047c0 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a08bfb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a040cc920 con 0x7f6a041594f0 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49364/0 (socket says 172.21.15.124:49364) 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 -- 172.21.15.124:0/64747225 learned_addr learned my addr 172.21.15.124:0/64747225 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a0259c640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 0x7f6a040cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 0x7f6a04159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 -- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 msgr2=0x7f6a04159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 0x7f6a04159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 -- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 msgr2=0x7f6a040cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 0x7f6a040cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a02d9d640 1 -- 172.21.15.124:0/64747225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a040ccc00 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a01d9b640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 0x7f6a04159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.458+0000 7f6a0259c640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 0x7f6a040cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f6a02d9d640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6a0408c510 tx=0x7f69fc011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc011ee0 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc004240 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a040ccee0 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc00f770 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.459+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a04159e70 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.460+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a04069400 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.460+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f69fc00f8d0 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.460+0000 7f69f2ffd640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 0x7f69e406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.461+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69fc089dd0 con 0x7f6a041f1b40 2023-12-13T23:56:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.461+0000 7f6a0259c640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 0x7f69e406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.461+0000 7f6a0259c640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 0x7f69e406d6b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007520 tx=0x7f69f400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.465+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f69fc054b90 con 0x7f6a041f1b40 2023-12-13T23:56:21.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.632+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a041605e0 con 0x7f6a041f1b40 2023-12-13T23:56:21.632 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.633+0000 7f69f2ffd640 1 -- 172.21.15.124:0/64747225 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f69fc019090 con 0x7f6a041f1b40 2023-12-13T23:56:21.632 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:21.632 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":35,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:19.985830+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 msgr2=0x7f69e406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 0x7f69e406d6b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f69f4007520 tx=0x7f69f400b040 comp rx=0 tx=0).stop 2023-12-13T23:56:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 msgr2=0x7f6a041f1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f6a0408c510 tx=0x7f69fc011cf0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 shutdown_connections 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f69e406b1f0 0x7f69e406d6b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6a041f1b40 0x7f6a041f1fc0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6a041594f0 0x7f6a04159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 --2- 172.21.15.124:0/64747225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f69ec0047c0 0x7f6a040cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 >> 172.21.15.124:0/64747225 conn(0x7f6a04072020 msgr2=0x7f6a0415e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 shutdown_connections 2023-12-13T23:56:21.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.635+0000 7f6a08bfb640 1 -- 172.21.15.124:0/64747225 wait complete. 2023-12-13T23:56:21.645 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-13T23:56:21.646 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:56:21.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.746+0000 7f5e996b2640 1 Processor -- start 2023-12-13T23:56:21.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.746+0000 7f5e996b2640 1 -- start start 2023-12-13T23:56:21.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e996b2640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5e940d2910 0x7f5e940d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e996b2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 0x7f5e940b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e996b2640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e940b2b10 con 0x7f5e9415d330 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e996b2640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e940d1590 con 0x7f5e940d2910 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e996b2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e940d1870 con 0x7f5e940d3730 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e92ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f5e9415d330 0x7f5e940b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36200/0 (socket says 172.21.15.124:36200) 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e92ffd640 1 -- 172.21.15.124:0/1350036608 learned_addr learned my addr 172.21.15.124:0/1350036608 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e937fe640 1 --2- 172.21.15.124:0/1350036608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 0x7f5e940b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e927fc640 1 --2- 172.21.15.124:0/1350036608 >> v2:172.21.15.203:3300/0 conn(0x7f5e940d2910 0x7f5e940d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994597789 0 0) 0x7f5e940b2b10 con 0x7f5e9415d330 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.747+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e7c003520 con 0x7f5e9415d330 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4037118979 0 0) 0x7f5e7c003520 con 0x7f5e9415d330 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e937fe640 1 -- 172.21.15.124:0/1350036608 >> v2:172.21.15.203:3300/0 conn(0x7f5e940d2910 msgr2=0x7f5e940d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e937fe640 1 --2- 172.21.15.124:0/1350036608 >> v2:172.21.15.203:3300/0 conn(0x7f5e940d2910 0x7f5e940d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e937fe640 1 -- 172.21.15.124:0/1350036608 >> v1:172.21.15.124:6789/0 conn(0x7f5e9415d330 legacy=0x7f5e940b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e937fe640 1 -- 172.21.15.124:0/1350036608 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e940d3b90 con 0x7f5e940d3730 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.748+0000 7f5e937fe640 1 --2- 172.21.15.124:0/1350036608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 0x7f5e940b6390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009d90 tx=0x7f5e8c02bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2eb0038210ad3cc4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84001670 con 0x7f5e940d3730 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8c02c810 con 0x7f5e940d3730 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e91ffb640 1 -- 172.21.15.124:0/1350036608 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8c02cb80 con 0x7f5e940d3730 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 -- 172.21.15.124:0/1350036608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 msgr2=0x7f5e940b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 --2- 172.21.15.124:0/1350036608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 0x7f5e940b6390 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5e8c009d90 tx=0x7f5e8c02bce0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 -- 172.21.15.124:0/1350036608 shutdown_connections 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 --2- 172.21.15.124:0/1350036608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e940d3730 0x7f5e940b6390 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 --2- 172.21.15.124:0/1350036608 >> v2:172.21.15.203:3300/0 conn(0x7f5e940d2910 0x7f5e940d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 -- 172.21.15.124:0/1350036608 >> 172.21.15.124:0/1350036608 conn(0x7f5e94072020 msgr2=0x7f5e94072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 -- 172.21.15.124:0/1350036608 shutdown_connections 2023-12-13T23:56:21.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.749+0000 7f5e996b2640 1 -- 172.21.15.124:0/1350036608 wait complete. 2023-12-13T23:56:21.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.750+0000 7f5e996b2640 1 Processor -- start 2023-12-13T23:56:21.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.750+0000 7f5e996b2640 1 -- start start 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 0x7f5e941501a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e9415d330 0x7f5e9415df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9415ec80 con 0x7f5e940d2910 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9415ef60 con 0x7f5e940d3730 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e996b2640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9415f240 con 0x7f5e9415d330 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e92ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e92ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49372/0 (socket says 172.21.15.124:49372) 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e92ffd640 1 -- 172.21.15.124:0/3450741123 learned_addr learned my addr 172.21.15.124:0/3450741123 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e937fe640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e9415d330 0x7f5e9415df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.751+0000 7f5e927fc640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 0x7f5e941501a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 -- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e9415d330 msgr2=0x7f5e9415df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e9415d330 0x7f5e9415df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 -- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 msgr2=0x7f5e941501a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 0x7f5e941501a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 -- 172.21.15.124:0/3450741123 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e8c009a70 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e927fc640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 0x7f5e941501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e92ffd640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c750 tx=0x7f5e8400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84005ad0 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e84005c30 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e9415f520 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e84007070 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.752+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e940b1890 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.753+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e9415fa40 con 0x7f5e940d2910 2023-12-13T23:56:21.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.753+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f5e840071d0 con 0x7f5e940d2910 2023-12-13T23:56:21.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.754+0000 7f5e7b7fe640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 0x7f5e8006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:21.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.754+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5e840897a0 con 0x7f5e940d2910 2023-12-13T23:56:21.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.754+0000 7f5e927fc640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 0x7f5e8006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:21.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.755+0000 7f5e927fc640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 0x7f5e8006d680 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e940d0930 tx=0x7f5e88008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:21.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.758+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5e9415fa40 con 0x7f5e940d2910 2023-12-13T23:56:21.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.920+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5e940b6c50 con 0x7f5e8006b1c0 2023-12-13T23:56:21.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.928+0000 7f5e7b7fe640 1 -- 172.21.15.124:0/3450741123 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+280001 (secure 0 0 0) 0x7f5e940b6c50 con 0x7f5e8006b1c0 2023-12-13T23:56:21.930 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:21.932 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 msgr2=0x7f5e8006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 0x7f5e8006d680 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5e940d0930 tx=0x7f5e88008040 comp rx=0 tx=0).stop 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 msgr2=0x7f5e940b1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e8400c750 tx=0x7f5e8400cc10 comp rx=0 tx=0).stop 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 shutdown_connections 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f5e8006b1c0 0x7f5e8006d680 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5e940d3730 0x7f5e941501a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f5e9415d330 0x7f5e9415df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.931+0000 7f5e996b2640 1 --2- 172.21.15.124:0/3450741123 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5e940d2910 0x7f5e940b1350 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.932+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 >> 172.21.15.124:0/3450741123 conn(0x7f5e94072020 msgr2=0x7f5e940cdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.932+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 shutdown_connections 2023-12-13T23:56:21.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:21.932+0000 7f5e996b2640 1 -- 172.21.15.124:0/3450741123 wait complete. 2023-12-13T23:56:21.941 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2023-12-13T23:56:20.125312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590472,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":76,"acting":76,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":20,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":220184,"kb_used_data":4012,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754536,"statfs":{"total":773094113280,"available":772868644864,"internally_reserved":0,"allocated":4108288,"data_stored":2295159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":104,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000928"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992925+0000","last_change":"2023-12-13T23:56:18.999424+0000","last_active":"2023-12-13T23:56:19.992925+0000","last_peered":"2023-12-13T23:56:19.992925+0000","last_clean":"2023-12-13T23:56:19.992925+0000","last_became_active":"2023-12-13T23:56:18.999108+0000","last_became_peered":"2023-12-13T23:56:18.999108+0000","last_unstale":"2023-12-13T23:56:19.992925+0000","last_undegraded":"2023-12-13T23:56:19.992925+0000","last_fullsized":"2023-12-13T23:56:19.992925+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:58.722565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993522+0000","last_change":"2023-12-13T23:56:19.000425+0000","last_active":"2023-12-13T23:56:19.993522+0000","last_peered":"2023-12-13T23:56:19.993522+0000","last_clean":"2023-12-13T23:56:19.993522+0000","last_became_active":"2023-12-13T23:56:19.000126+0000","last_became_peered":"2023-12-13T23:56:19.000126+0000","last_unstale":"2023-12-13T23:56:19.993522+0000","last_undegraded":"2023-12-13T23:56:19.993522+0000","last_fullsized":"2023-12-13T23:56:19.993522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:06:02.930985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993360+0000","last_change":"2023-12-13T23:56:19.005760+0000","last_active":"2023-12-13T23:56:19.993360+0000","last_peered":"2023-12-13T23:56:19.993360+0000","last_clean":"2023-12-13T23:56:19.993360+0000","last_became_active":"2023-12-13T23:56:19.005463+0000","last_became_peered":"2023-12-13T23:56:19.005463+0000","last_unstale":"2023-12-13T23:56:19.993360+0000","last_undegraded":"2023-12-13T23:56:19.993360+0000","last_fullsized":"2023-12-13T23:56:19.993360+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:55.752709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993803+0000","last_change":"2023-12-13T23:56:19.004995+0000","last_active":"2023-12-13T23:56:19.993803+0000","last_peered":"2023-12-13T23:56:19.993803+0000","last_clean":"2023-12-13T23:56:19.993803+0000","last_became_active":"2023-12-13T23:56:19.004591+0000","last_became_peered":"2023-12-13T23:56:19.004591+0000","last_unstale":"2023-12-13T23:56:19.993803+0000","last_undegraded":"2023-12-13T23:56:19.993803+0000","last_fullsized":"2023-12-13T23:56:19.993803+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:53:03.806070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993777+0000","last_change":"2023-12-13T23:56:19.004339+0000","last_active":"2023-12-13T23:56:19.993777+0000","last_peered":"2023-12-13T23:56:19.993777+0000","last_clean":"2023-12-13T23:56:19.993777+0000","last_became_active":"2023-12-13T23:56:19.004017+0000","last_became_peered":"2023-12-13T23:56:19.004017+0000","last_unstale":"2023-12-13T23:56:19.993777+0000","last_undegraded":"2023-12-13T23:56:19.993777+0000","last_fullsized":"2023-12-13T23:56:19.993777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:07:28.143709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993803+0000","last_change":"2023-12-13T23:56:18.999020+0000","last_active":"2023-12-13T23:56:19.993803+0000","last_peered":"2023-12-13T23:56:19.993803+0000","last_clean":"2023-12-13T23:56:19.993803+0000","last_became_active":"2023-12-13T23:56:18.998712+0000","last_became_peered":"2023-12-13T23:56:18.998712+0000","last_unstale":"2023-12-13T23:56:19.993803+0000","last_undegraded":"2023-12-13T23:56:19.993803+0000","last_fullsized":"2023-12-13T23:56:19.993803+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:24:32.727328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992466+0000","last_change":"2023-12-13T23:56:19.008649+0000","last_active":"2023-12-13T23:56:19.992466+0000","last_peered":"2023-12-13T23:56:19.992466+0000","last_clean":"2023-12-13T23:56:19.992466+0000","last_became_active":"2023-12-13T23:56:19.006114+0000","last_became_peered":"2023-12-13T23:56:19.006114+0000","last_unstale":"2023-12-13T23:56:19.992466+0000","last_undegraded":"2023-12-13T23:56:19.992466+0000","last_fullsized":"2023-12-13T23:56:19.992466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:15:31.032348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992506+0000","last_change":"2023-12-13T23:56:19.006147+0000","last_active":"2023-12-13T23:56:19.992506+0000","last_peered":"2023-12-13T23:56:19.992506+0000","last_clean":"2023-12-13T23:56:19.992506+0000","last_became_active":"2023-12-13T23:56:19.005834+0000","last_became_peered":"2023-12-13T23:56:19.005834+0000","last_unstale":"2023-12-13T23:56:19.992506+0000","last_undegraded":"2023-12-13T23:56:19.992506+0000","last_fullsized":"2023-12-13T23:56:19.992506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:08:26.045928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.004761+0000","last_change":"2023-12-13T23:56:19.003956+0000","last_active":"2023-12-13T23:56:19.004761+0000","last_peered":"2023-12-13T23:56:19.004761+0000","last_clean":"2023-12-13T23:56:19.004761+0000","last_became_active":"2023-12-13T23:56:19.003712+0000","last_became_peered":"2023-12-13T23:56:19.003712+0000","last_unstale":"2023-12-13T23:56:19.004761+0000","last_undegraded":"2023-12-13T23:56:19.004761+0000","last_fullsized":"2023-12-13T23:56:19.004761+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:55.196668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992943+0000","last_change":"2023-12-13T23:56:19.001641+0000","last_active":"2023-12-13T23:56:19.992943+0000","last_peered":"2023-12-13T23:56:19.992943+0000","last_clean":"2023-12-13T23:56:19.992943+0000","last_became_active":"2023-12-13T23:56:19.001390+0000","last_became_peered":"2023-12-13T23:56:19.001390+0000","last_unstale":"2023-12-13T23:56:19.992943+0000","last_undegraded":"2023-12-13T23:56:19.992943+0000","last_fullsized":"2023-12-13T23:56:19.992943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:39:54.581072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993257+0000","last_change":"2023-12-13T23:56:18.999499+0000","last_active":"2023-12-13T23:56:19.993257+0000","last_peered":"2023-12-13T23:56:19.993257+0000","last_clean":"2023-12-13T23:56:19.993257+0000","last_became_active":"2023-12-13T23:56:18.998668+0000","last_became_peered":"2023-12-13T23:56:18.998668+0000","last_unstale":"2023-12-13T23:56:19.993257+0000","last_undegraded":"2023-12-13T23:56:19.993257+0000","last_fullsized":"2023-12-13T23:56:19.993257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:29:26.833083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993215+0000","last_change":"2023-12-13T23:56:19.009981+0000","last_active":"2023-12-13T23:56:19.993215+0000","last_peered":"2023-12-13T23:56:19.993215+0000","last_clean":"2023-12-13T23:56:19.993215+0000","last_became_active":"2023-12-13T23:56:19.009722+0000","last_became_peered":"2023-12-13T23:56:19.009722+0000","last_unstale":"2023-12-13T23:56:19.993215+0000","last_undegraded":"2023-12-13T23:56:19.993215+0000","last_fullsized":"2023-12-13T23:56:19.993215+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:59:36.293125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007784+0000","last_change":"2023-12-13T23:56:19.005988+0000","last_active":"2023-12-13T23:56:19.007784+0000","last_peered":"2023-12-13T23:56:19.007784+0000","last_clean":"2023-12-13T23:56:19.007784+0000","last_became_active":"2023-12-13T23:56:19.005737+0000","last_became_peered":"2023-12-13T23:56:19.005737+0000","last_unstale":"2023-12-13T23:56:19.007784+0000","last_undegraded":"2023-12-13T23:56:19.007784+0000","last_fullsized":"2023-12-13T23:56:19.007784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:39.201454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.009169+0000","last_change":"2023-12-13T23:56:17.984701+0000","last_active":"2023-12-13T23:56:19.009169+0000","last_peered":"2023-12-13T23:56:19.009169+0000","last_clean":"2023-12-13T23:56:19.009169+0000","last_became_active":"2023-12-13T23:56:17.984452+0000","last_became_peered":"2023-12-13T23:56:17.984452+0000","last_unstale":"2023-12-13T23:56:19.009169+0000","last_undegraded":"2023-12-13T23:56:19.009169+0000","last_fullsized":"2023-12-13T23:56:19.009169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:20.932255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993425+0000","last_change":"2023-12-13T23:56:19.011597+0000","last_active":"2023-12-13T23:56:19.993425+0000","last_peered":"2023-12-13T23:56:19.993425+0000","last_clean":"2023-12-13T23:56:19.993425+0000","last_became_active":"2023-12-13T23:56:19.011208+0000","last_became_peered":"2023-12-13T23:56:19.011208+0000","last_unstale":"2023-12-13T23:56:19.993425+0000","last_undegraded":"2023-12-13T23:56:19.993425+0000","last_fullsized":"2023-12-13T23:56:19.993425+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:58.722565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993685+0000","last_change":"2023-12-13T23:56:19.010582+0000","last_active":"2023-12-13T23:56:19.993685+0000","last_peered":"2023-12-13T23:56:19.993685+0000","last_clean":"2023-12-13T23:56:19.993685+0000","last_became_active":"2023-12-13T23:56:19.010250+0000","last_became_peered":"2023-12-13T23:56:19.010250+0000","last_unstale":"2023-12-13T23:56:19.993685+0000","last_undegraded":"2023-12-13T23:56:19.993685+0000","last_fullsized":"2023-12-13T23:56:19.993685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:11:25.231381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992700+0000","last_change":"2023-12-13T23:56:17.993548+0000","last_active":"2023-12-13T23:56:19.992700+0000","last_peered":"2023-12-13T23:56:19.992700+0000","last_clean":"2023-12-13T23:56:19.992700+0000","last_became_active":"2023-12-13T23:56:17.992478+0000","last_became_peered":"2023-12-13T23:56:17.992478+0000","last_unstale":"2023-12-13T23:56:19.992700+0000","last_undegraded":"2023-12-13T23:56:19.992700+0000","last_fullsized":"2023-12-13T23:56:19.992700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:15.463433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992644+0000","last_change":"2023-12-13T23:56:17.992714+0000","last_active":"2023-12-13T23:56:19.992644+0000","last_peered":"2023-12-13T23:56:19.992644+0000","last_clean":"2023-12-13T23:56:19.992644+0000","last_became_active":"2023-12-13T23:56:17.992288+0000","last_became_peered":"2023-12-13T23:56:17.992288+0000","last_unstale":"2023-12-13T23:56:19.992644+0000","last_undegraded":"2023-12-13T23:56:19.992644+0000","last_fullsized":"2023-12-13T23:56:19.992644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:38.197220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992361+0000","last_change":"2023-12-13T23:56:18.998439+0000","last_active":"2023-12-13T23:56:19.992361+0000","last_peered":"2023-12-13T23:56:19.992361+0000","last_clean":"2023-12-13T23:56:19.992361+0000","last_became_active":"2023-12-13T23:56:18.998168+0000","last_became_peered":"2023-12-13T23:56:18.998168+0000","last_unstale":"2023-12-13T23:56:19.992361+0000","last_undegraded":"2023-12-13T23:56:19.992361+0000","last_fullsized":"2023-12-13T23:56:19.992361+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:07:16.341679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992468+0000","last_change":"2023-12-13T23:56:17.997038+0000","last_active":"2023-12-13T23:56:19.992468+0000","last_peered":"2023-12-13T23:56:19.992468+0000","last_clean":"2023-12-13T23:56:19.992468+0000","last_became_active":"2023-12-13T23:56:17.996755+0000","last_became_peered":"2023-12-13T23:56:17.996755+0000","last_unstale":"2023-12-13T23:56:19.992468+0000","last_undegraded":"2023-12-13T23:56:19.992468+0000","last_fullsized":"2023-12-13T23:56:19.992468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-14T23:57:31.262514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.997947+0000","last_change":"2023-12-13T23:56:17.991227+0000","last_active":"2023-12-13T23:56:18.997947+0000","last_peered":"2023-12-13T23:56:18.997947+0000","last_clean":"2023-12-13T23:56:18.997947+0000","last_became_active":"2023-12-13T23:56:17.990949+0000","last_became_peered":"2023-12-13T23:56:17.990949+0000","last_unstale":"2023-12-13T23:56:18.997947+0000","last_undegraded":"2023-12-13T23:56:18.997947+0000","last_fullsized":"2023-12-13T23:56:18.997947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:12:14.164111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993045+0000","last_change":"2023-12-13T23:56:17.992938+0000","last_active":"2023-12-13T23:56:19.993045+0000","last_peered":"2023-12-13T23:56:19.993045+0000","last_clean":"2023-12-13T23:56:19.993045+0000","last_became_active":"2023-12-13T23:56:17.992632+0000","last_became_peered":"2023-12-13T23:56:17.992632+0000","last_unstale":"2023-12-13T23:56:19.993045+0000","last_undegraded":"2023-12-13T23:56:19.993045+0000","last_fullsized":"2023-12-13T23:56:19.993045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:29.760814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993481+0000","last_change":"2023-12-13T23:56:19.009375+0000","last_active":"2023-12-13T23:56:19.993481+0000","last_peered":"2023-12-13T23:56:19.993481+0000","last_clean":"2023-12-13T23:56:19.993481+0000","last_became_active":"2023-12-13T23:56:19.009108+0000","last_became_peered":"2023-12-13T23:56:19.009108+0000","last_unstale":"2023-12-13T23:56:19.993481+0000","last_undegraded":"2023-12-13T23:56:19.993481+0000","last_fullsized":"2023-12-13T23:56:19.993481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:17:58.669960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993157+0000","last_change":"2023-12-13T23:56:19.009198+0000","last_active":"2023-12-13T23:56:19.993157+0000","last_peered":"2023-12-13T23:56:19.993157+0000","last_clean":"2023-12-13T23:56:19.993157+0000","last_became_active":"2023-12-13T23:56:19.008877+0000","last_became_peered":"2023-12-13T23:56:19.008877+0000","last_unstale":"2023-12-13T23:56:19.993157+0000","last_undegraded":"2023-12-13T23:56:19.993157+0000","last_fullsized":"2023-12-13T23:56:19.993157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:57:12.686117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993246+0000","last_change":"2023-12-13T23:56:17.992825+0000","last_active":"2023-12-13T23:56:19.993246+0000","last_peered":"2023-12-13T23:56:19.993246+0000","last_clean":"2023-12-13T23:56:19.993246+0000","last_became_active":"2023-12-13T23:56:17.992527+0000","last_became_peered":"2023-12-13T23:56:17.992527+0000","last_unstale":"2023-12-13T23:56:19.993246+0000","last_undegraded":"2023-12-13T23:56:19.993246+0000","last_fullsized":"2023-12-13T23:56:19.993246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:54.192433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993073+0000","last_change":"2023-12-13T23:56:17.993522+0000","last_active":"2023-12-13T23:56:19.993073+0000","last_peered":"2023-12-13T23:56:19.993073+0000","last_clean":"2023-12-13T23:56:19.993073+0000","last_became_active":"2023-12-13T23:56:17.993228+0000","last_became_peered":"2023-12-13T23:56:17.993228+0000","last_unstale":"2023-12-13T23:56:19.993073+0000","last_undegraded":"2023-12-13T23:56:19.993073+0000","last_fullsized":"2023-12-13T23:56:19.993073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:25:36.100047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992880+0000","last_change":"2023-12-13T23:56:17.992764+0000","last_active":"2023-12-13T23:56:19.992880+0000","last_peered":"2023-12-13T23:56:19.992880+0000","last_clean":"2023-12-13T23:56:19.992880+0000","last_became_active":"2023-12-13T23:56:17.992485+0000","last_became_peered":"2023-12-13T23:56:17.992485+0000","last_unstale":"2023-12-13T23:56:19.992880+0000","last_undegraded":"2023-12-13T23:56:19.992880+0000","last_fullsized":"2023-12-13T23:56:19.992880+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:20.932255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.009717+0000","last_change":"2023-12-13T23:56:17.984295+0000","last_active":"2023-12-13T23:56:19.009717+0000","last_peered":"2023-12-13T23:56:19.009717+0000","last_clean":"2023-12-13T23:56:19.009717+0000","last_became_active":"2023-12-13T23:56:17.983873+0000","last_became_peered":"2023-12-13T23:56:17.983873+0000","last_unstale":"2023-12-13T23:56:19.009717+0000","last_undegraded":"2023-12-13T23:56:19.009717+0000","last_fullsized":"2023-12-13T23:56:19.009717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:29.760814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992873+0000","last_change":"2023-12-13T23:56:17.996506+0000","last_active":"2023-12-13T23:56:19.992873+0000","last_peered":"2023-12-13T23:56:19.992873+0000","last_clean":"2023-12-13T23:56:19.992873+0000","last_became_active":"2023-12-13T23:56:17.996184+0000","last_became_peered":"2023-12-13T23:56:17.996184+0000","last_unstale":"2023-12-13T23:56:19.992873+0000","last_undegraded":"2023-12-13T23:56:19.992873+0000","last_fullsized":"2023-12-13T23:56:19.992873+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:37.013270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992719+0000","last_change":"2023-12-13T23:56:18.998532+0000","last_active":"2023-12-13T23:56:19.992719+0000","last_peered":"2023-12-13T23:56:19.992719+0000","last_clean":"2023-12-13T23:56:19.992719+0000","last_became_active":"2023-12-13T23:56:18.998279+0000","last_became_peered":"2023-12-13T23:56:18.998279+0000","last_unstale":"2023-12-13T23:56:19.992719+0000","last_undegraded":"2023-12-13T23:56:19.992719+0000","last_fullsized":"2023-12-13T23:56:19.992719+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:28.726612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.998069+0000","last_change":"2023-12-13T23:56:17.987634+0000","last_active":"2023-12-13T23:56:18.998069+0000","last_peered":"2023-12-13T23:56:18.998069+0000","last_clean":"2023-12-13T23:56:18.998069+0000","last_became_active":"2023-12-13T23:56:17.987398+0000","last_became_peered":"2023-12-13T23:56:17.987398+0000","last_unstale":"2023-12-13T23:56:18.998069+0000","last_undegraded":"2023-12-13T23:56:18.998069+0000","last_fullsized":"2023-12-13T23:56:18.998069+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:15.463433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006566+0000","last_change":"2023-12-13T23:56:19.004882+0000","last_active":"2023-12-13T23:56:19.006566+0000","last_peered":"2023-12-13T23:56:19.006566+0000","last_clean":"2023-12-13T23:56:19.006566+0000","last_became_active":"2023-12-13T23:56:19.004623+0000","last_became_peered":"2023-12-13T23:56:19.004623+0000","last_unstale":"2023-12-13T23:56:19.006566+0000","last_undegraded":"2023-12-13T23:56:19.006566+0000","last_fullsized":"2023-12-13T23:56:19.006566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:25:01.043631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.998429+0000","last_change":"2023-12-13T23:56:17.986688+0000","last_active":"2023-12-13T23:56:18.998429+0000","last_peered":"2023-12-13T23:56:18.998429+0000","last_clean":"2023-12-13T23:56:18.998429+0000","last_became_active":"2023-12-13T23:56:17.986318+0000","last_became_peered":"2023-12-13T23:56:17.986318+0000","last_unstale":"2023-12-13T23:56:18.998429+0000","last_undegraded":"2023-12-13T23:56:18.998429+0000","last_fullsized":"2023-12-13T23:56:18.998429+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:18:21.702239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.008926+0000","last_change":"2023-12-13T23:56:19.005583+0000","last_active":"2023-12-13T23:56:19.008926+0000","last_peered":"2023-12-13T23:56:19.008926+0000","last_clean":"2023-12-13T23:56:19.008926+0000","last_became_active":"2023-12-13T23:56:19.005242+0000","last_became_peered":"2023-12-13T23:56:19.005242+0000","last_unstale":"2023-12-13T23:56:19.008926+0000","last_undegraded":"2023-12-13T23:56:19.008926+0000","last_fullsized":"2023-12-13T23:56:19.008926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:44:39.483806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:17.970612+0000","last_change":"2023-12-13T23:56:17.970612+0000","last_active":"2023-12-13T23:56:17.970612+0000","last_peered":"2023-12-13T23:56:17.970612+0000","last_clean":"2023-12-13T23:56:17.970612+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:17.970612+0000","last_undegraded":"2023-12-13T23:56:17.970612+0000","last_fullsized":"2023-12-13T23:56:17.970612+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992964+0000","last_change":"2023-12-13T23:56:17.994238+0000","last_active":"2023-12-13T23:56:19.992964+0000","last_peered":"2023-12-13T23:56:19.992964+0000","last_clean":"2023-12-13T23:56:19.992964+0000","last_became_active":"2023-12-13T23:56:17.993973+0000","last_became_peered":"2023-12-13T23:56:17.993973+0000","last_unstale":"2023-12-13T23:56:19.992964+0000","last_undegraded":"2023-12-13T23:56:19.992964+0000","last_fullsized":"2023-12-13T23:56:19.992964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:41:05.818945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993372+0000","last_change":"2023-12-13T23:56:19.002051+0000","last_active":"2023-12-13T23:56:19.993372+0000","last_peered":"2023-12-13T23:56:19.993372+0000","last_clean":"2023-12-13T23:56:19.993372+0000","last_became_active":"2023-12-13T23:56:19.001761+0000","last_became_peered":"2023-12-13T23:56:19.001761+0000","last_unstale":"2023-12-13T23:56:19.993372+0000","last_undegraded":"2023-12-13T23:56:19.993372+0000","last_fullsized":"2023-12-13T23:56:19.993372+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:28.726612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992457+0000","last_change":"2023-12-13T23:56:19.006703+0000","last_active":"2023-12-13T23:56:19.992457+0000","last_peered":"2023-12-13T23:56:19.992457+0000","last_clean":"2023-12-13T23:56:19.992457+0000","last_became_active":"2023-12-13T23:56:19.006443+0000","last_became_peered":"2023-12-13T23:56:19.006443+0000","last_unstale":"2023-12-13T23:56:19.992457+0000","last_undegraded":"2023-12-13T23:56:19.992457+0000","last_fullsized":"2023-12-13T23:56:19.992457+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:55.752709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-12-13T23:56:16.966377+0000","last_change":"2023-12-13T23:56:16.966377+0000","last_active":"2023-12-13T23:56:16.966377+0000","last_peered":"2023-12-13T23:56:16.966377+0000","last_clean":"2023-12-13T23:56:16.966377+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-12-13T23:56:16.966377+0000","last_undegraded":"2023-12-13T23:56:16.966377+0000","last_fullsized":"2023-12-13T23:56:16.966377+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.992267+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:56:18.992267+0000","last_peered":"2023-12-13T23:56:18.992267+0000","last_clean":"2023-12-13T23:56:18.992267+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:56:18.992267+0000","last_undegraded":"2023-12-13T23:56:18.992267+0000","last_fullsized":"2023-12-13T23:56:18.992267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":46,"acting":46,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":104,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":28,"acting":28,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":747036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.75},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":131419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.442},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.69399999999999995}]},{"osd":3,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.5190000000000001,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.58699999999999997}]},{"osd":4,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.755}]},{"osd":5,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63400000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.67100000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":131419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.55400000000000005}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.505},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.51400000000000001}]},{"osd":2,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57199999999999995}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47799999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.41199999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.38600000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.42599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":150382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27364,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":348160,"data_stored":131419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"last":0.29699999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"last":0.314}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.35299999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.73499999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.68500000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.64800000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56899999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.316},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.38500000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.52900000000000003}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.50800000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.79900000000000004}]},{"osd":5,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.89900000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.84199999999999997}]},{"osd":7,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.74399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":131419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.40699999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.435},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.38800000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.61499999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.54800000000000004}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.63},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.52700000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.65300000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":740646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.316}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.377}]},{"osd":3,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.36399999999999999}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.63}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.56999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:56:21.948 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:56:21.957 INFO:tasks.ceph.mds.a.smithi124.stderr:2023-12-13T23:56:21.958+0000 7f1961a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:56:22.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.058+0000 7fd300675640 1 Processor -- start 2023-12-13T23:56:22.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.059+0000 7fd300675640 1 -- start start 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.059+0000 7fd300675640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.059+0000 7fd300675640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd2f80d3730 0x7fd2f80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.059+0000 7fd300675640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f80b2b10 con 0x7fd2f815d330 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd300675640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80d3b50 con 0x7fd2f80d3730 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd300675640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80d1590 con 0x7fd2f80d2910 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fe411640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd2f815d330 0x7fd2f80b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36206/0 (socket says 172.21.15.124:36206) 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fe411640 1 -- 172.21.15.124:0/432398026 learned_addr learned my addr 172.21.15.124:0/432398026 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fec12640 1 --2- 172.21.15.124:0/432398026 >> v2:172.21.15.203:3300/0 conn(0x7fd2f80d3730 0x7fd2f80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fdc10640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fd40f640 1 -- 172.21.15.124:0/432398026 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2222039631 0 0) 0x7fd2f80b2b10 con 0x7fd2f815d330 2023-12-13T23:56:22.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fd40f640 1 -- 172.21.15.124:0/432398026 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2e4003520 con 0x7fd2f815d330 2023-12-13T23:56:22.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fec12640 1 -- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 msgr2=0x7fd2f80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fec12640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fec12640 1 -- 172.21.15.124:0/432398026 >> v1:172.21.15.124:6789/0 conn(0x7fd2f815d330 legacy=0x7fd2f80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.060+0000 7fd2fec12640 1 -- 172.21.15.124:0/432398026 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80d1870 con 0x7fd2f80d3730 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.061+0000 7fd2fec12640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f80d3730 0x7fd2f80b6390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d90 tx=0x7fd2f002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8168df1212b1add server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.061+0000 7fd2fd40f640 1 -- 172.21.15.124:0/432398026 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f002cbc0 con 0x7fd2f80d3730 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.061+0000 7fd2fd40f640 1 -- 172.21.15.124:0/432398026 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f002cd20 con 0x7fd2f80d3730 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.061+0000 7fd2fd40f640 1 -- 172.21.15.124:0/432398026 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f0034660 con 0x7fd2f80d3730 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 -- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f80d3730 msgr2=0x7fd2f80b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f80d3730 0x7fd2f80b6390 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d90 tx=0x7fd2f002be10 comp rx=0 tx=0).stop 2023-12-13T23:56:22.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 -- 172.21.15.124:0/432398026 shutdown_connections 2023-12-13T23:56:22.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f80d3730 0x7fd2f80b6390 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 --2- 172.21.15.124:0/432398026 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 -- 172.21.15.124:0/432398026 >> 172.21.15.124:0/432398026 conn(0x7fd2f8072020 msgr2=0x7fd2f8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 -- 172.21.15.124:0/432398026 shutdown_connections 2023-12-13T23:56:22.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.062+0000 7fd300675640 1 -- 172.21.15.124:0/432398026 wait complete. 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.063+0000 7fd300675640 1 Processor -- start 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.063+0000 7fd300675640 1 -- start start 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f815d330 0x7fd2f80ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81f2e70 con 0x7fd2f80d3730 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c82f0 con 0x7fd2f815d330 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c85d0 con 0x7fd2f80d2910 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49384/0 (socket says 172.21.15.124:49384) 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 -- 172.21.15.124:0/1522481831 learned_addr learned my addr 172.21.15.124:0/1522481831 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fec12640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f815d330 0x7fd2f80ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fe411640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 -- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 msgr2=0x7fd2f80c7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80c7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 -- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f815d330 msgr2=0x7fd2f80ca870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f815d330 0x7fd2f80ca870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 -- 172.21.15.124:0/1522481831 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0009a70 con 0x7fd2f80d3730 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2fdc10640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e870 tx=0x7fd2f400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f40094f0 con 0x7fd2f80d3730 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f400c940 con 0x7fd2f80d3730 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f80c88b0 con 0x7fd2f80d3730 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f400ccf0 con 0x7fd2f80d3730 2023-12-13T23:56:22.065 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.064+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f80c8e70 con 0x7fd2f80d3730 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.065+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f8069400 con 0x7fd2f80d3730 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.066+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd2f4013470 con 0x7fd2f80d3730 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.066+0000 7fd2eeffd640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 0x7fd2dc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.066+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2f4089a90 con 0x7fd2f80d3730 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.067+0000 7fd2fe411640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 0x7fd2dc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.067+0000 7fd2fe411640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 0x7fd2dc06d680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80011e0 tx=0x7fd2e8007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.071+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd2f4054870 con 0x7fd2f80d3730 2023-12-13T23:56:22.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.235+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2f80c8740 con 0x7fd2dc06b1c0 2023-12-13T23:56:22.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.242+0000 7fd2eeffd640 1 -- 172.21.15.124:0/1522481831 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289020 (secure 0 0 0) 0x7fd2f80c8740 con 0x7fd2dc06b1c0 2023-12-13T23:56:22.244 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:22.246 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:56:22.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.245+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 msgr2=0x7fd2dc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.245+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 0x7fd2dc06d680 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd2e80011e0 tx=0x7fd2e8007d30 comp rx=0 tx=0).stop 2023-12-13T23:56:22.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 msgr2=0x7fd2f80ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400e870 tx=0x7fd2f400ed30 comp rx=0 tx=0).stop 2023-12-13T23:56:22.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 shutdown_connections 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd2dc06b1c0 0x7fd2dc06d680 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd2f815d330 0x7fd2f80ca870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd2f80d3730 0x7fd2f80ca330 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 --2- 172.21.15.124:0/1522481831 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd2f80d2910 0x7fd2f80c7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 >> 172.21.15.124:0/1522481831 conn(0x7fd2f8072020 msgr2=0x7fd2f80b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 shutdown_connections 2023-12-13T23:56:22.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.246+0000 7fd300675640 1 -- 172.21.15.124:0/1522481831 wait complete. 2023-12-13T23:56:22.255 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2023-12-13T23:56:22.125751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":97,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220404,"kb_used_data":4244,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754316,"statfs":{"total":773094113280,"available":772868419584,"internally_reserved":0,"allocated":4345856,"data_stored":2394080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.154346"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.004819+0000","last_change":"2023-12-13T23:56:18.999626+0000","last_active":"2023-12-13T23:56:19.004819+0000","last_peered":"2023-12-13T23:56:19.004819+0000","last_clean":"2023-12-13T23:56:19.004819+0000","last_became_active":"2023-12-13T23:56:18.999336+0000","last_became_peered":"2023-12-13T23:56:18.999336+0000","last_unstale":"2023-12-13T23:56:19.004819+0000","last_undegraded":"2023-12-13T23:56:19.004819+0000","last_fullsized":"2023-12-13T23:56:19.004819+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:44:39.483806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013405+0000","last_change":"2023-12-13T23:56:19.008207+0000","last_active":"2023-12-13T23:56:19.013405+0000","last_peered":"2023-12-13T23:56:19.013405+0000","last_clean":"2023-12-13T23:56:19.013405+0000","last_became_active":"2023-12-13T23:56:19.007926+0000","last_became_peered":"2023-12-13T23:56:19.007926+0000","last_unstale":"2023-12-13T23:56:19.013405+0000","last_undegraded":"2023-12-13T23:56:19.013405+0000","last_fullsized":"2023-12-13T23:56:19.013405+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:06:02.930985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.010980+0000","last_change":"2023-12-13T23:56:19.006539+0000","last_active":"2023-12-13T23:56:19.010980+0000","last_peered":"2023-12-13T23:56:19.010980+0000","last_clean":"2023-12-13T23:56:19.010980+0000","last_became_active":"2023-12-13T23:56:19.005138+0000","last_became_peered":"2023-12-13T23:56:19.005138+0000","last_unstale":"2023-12-13T23:56:19.010980+0000","last_undegraded":"2023-12-13T23:56:19.010980+0000","last_fullsized":"2023-12-13T23:56:19.010980+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:58.722565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.015621+0000","last_change":"2023-12-13T23:56:19.011536+0000","last_active":"2023-12-13T23:56:19.015621+0000","last_peered":"2023-12-13T23:56:19.015621+0000","last_clean":"2023-12-13T23:56:19.015621+0000","last_became_active":"2023-12-13T23:56:19.010817+0000","last_became_peered":"2023-12-13T23:56:19.010817+0000","last_unstale":"2023-12-13T23:56:19.015621+0000","last_undegraded":"2023-12-13T23:56:19.015621+0000","last_fullsized":"2023-12-13T23:56:19.015621+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:16.467667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.988963+0000","last_change":"2023-12-13T23:56:19.010180+0000","last_active":"2023-12-13T23:56:19.988963+0000","last_peered":"2023-12-13T23:56:19.988963+0000","last_clean":"2023-12-13T23:56:19.988963+0000","last_became_active":"2023-12-13T23:56:19.009909+0000","last_became_peered":"2023-12-13T23:56:19.009909+0000","last_unstale":"2023-12-13T23:56:19.988963+0000","last_undegraded":"2023-12-13T23:56:19.988963+0000","last_fullsized":"2023-12-13T23:56:19.988963+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:38.017505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.004602+0000","last_change":"2023-12-13T23:56:18.999689+0000","last_active":"2023-12-13T23:56:19.004602+0000","last_peered":"2023-12-13T23:56:19.004602+0000","last_clean":"2023-12-13T23:56:19.004602+0000","last_became_active":"2023-12-13T23:56:18.999436+0000","last_became_peered":"2023-12-13T23:56:18.999436+0000","last_unstale":"2023-12-13T23:56:19.004602+0000","last_undegraded":"2023-12-13T23:56:19.004602+0000","last_fullsized":"2023-12-13T23:56:19.004602+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:18:22.706473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992925+0000","last_change":"2023-12-13T23:56:18.999424+0000","last_active":"2023-12-13T23:56:19.992925+0000","last_peered":"2023-12-13T23:56:19.992925+0000","last_clean":"2023-12-13T23:56:19.992925+0000","last_became_active":"2023-12-13T23:56:18.999108+0000","last_became_peered":"2023-12-13T23:56:18.999108+0000","last_unstale":"2023-12-13T23:56:19.992925+0000","last_undegraded":"2023-12-13T23:56:19.992925+0000","last_fullsized":"2023-12-13T23:56:19.992925+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:58.722565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993522+0000","last_change":"2023-12-13T23:56:19.000425+0000","last_active":"2023-12-13T23:56:19.993522+0000","last_peered":"2023-12-13T23:56:19.993522+0000","last_clean":"2023-12-13T23:56:19.993522+0000","last_became_active":"2023-12-13T23:56:19.000126+0000","last_became_peered":"2023-12-13T23:56:19.000126+0000","last_unstale":"2023-12-13T23:56:19.993522+0000","last_undegraded":"2023-12-13T23:56:19.993522+0000","last_fullsized":"2023-12-13T23:56:19.993522+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:06:02.930985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993360+0000","last_change":"2023-12-13T23:56:19.005760+0000","last_active":"2023-12-13T23:56:19.993360+0000","last_peered":"2023-12-13T23:56:19.993360+0000","last_clean":"2023-12-13T23:56:19.993360+0000","last_became_active":"2023-12-13T23:56:19.005463+0000","last_became_peered":"2023-12-13T23:56:19.005463+0000","last_unstale":"2023-12-13T23:56:19.993360+0000","last_undegraded":"2023-12-13T23:56:19.993360+0000","last_fullsized":"2023-12-13T23:56:19.993360+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:55.752709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013844+0000","last_change":"2023-12-13T23:56:19.009358+0000","last_active":"2023-12-13T23:56:19.013844+0000","last_peered":"2023-12-13T23:56:19.013844+0000","last_clean":"2023-12-13T23:56:19.013844+0000","last_became_active":"2023-12-13T23:56:19.008614+0000","last_became_peered":"2023-12-13T23:56:19.008614+0000","last_unstale":"2023-12-13T23:56:19.013844+0000","last_undegraded":"2023-12-13T23:56:19.013844+0000","last_fullsized":"2023-12-13T23:56:19.013844+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:12:15.168346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993803+0000","last_change":"2023-12-13T23:56:19.004995+0000","last_active":"2023-12-13T23:56:19.993803+0000","last_peered":"2023-12-13T23:56:19.993803+0000","last_clean":"2023-12-13T23:56:19.993803+0000","last_became_active":"2023-12-13T23:56:19.004591+0000","last_became_peered":"2023-12-13T23:56:19.004591+0000","last_unstale":"2023-12-13T23:56:19.993803+0000","last_undegraded":"2023-12-13T23:56:19.993803+0000","last_fullsized":"2023-12-13T23:56:19.993803+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:53:03.806070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.003160+0000","last_change":"2023-12-13T23:56:19.000827+0000","last_active":"2023-12-13T23:56:19.003160+0000","last_peered":"2023-12-13T23:56:19.003160+0000","last_clean":"2023-12-13T23:56:19.003160+0000","last_became_active":"2023-12-13T23:56:19.000582+0000","last_became_peered":"2023-12-13T23:56:19.000582+0000","last_unstale":"2023-12-13T23:56:19.003160+0000","last_undegraded":"2023-12-13T23:56:19.003160+0000","last_fullsized":"2023-12-13T23:56:19.003160+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:08:26.045928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.989152+0000","last_change":"2023-12-13T23:56:19.004169+0000","last_active":"2023-12-13T23:56:19.989152+0000","last_peered":"2023-12-13T23:56:19.989152+0000","last_clean":"2023-12-13T23:56:19.989152+0000","last_became_active":"2023-12-13T23:56:19.003925+0000","last_became_peered":"2023-12-13T23:56:19.003925+0000","last_unstale":"2023-12-13T23:56:19.989152+0000","last_undegraded":"2023-12-13T23:56:19.989152+0000","last_fullsized":"2023-12-13T23:56:19.989152+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:44:39.483806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006267+0000","last_change":"2023-12-13T23:56:19.003305+0000","last_active":"2023-12-13T23:56:19.006267+0000","last_peered":"2023-12-13T23:56:19.006267+0000","last_clean":"2023-12-13T23:56:19.006267+0000","last_became_active":"2023-12-13T23:56:19.003047+0000","last_became_peered":"2023-12-13T23:56:19.003047+0000","last_unstale":"2023-12-13T23:56:19.006267+0000","last_undegraded":"2023-12-13T23:56:19.006267+0000","last_fullsized":"2023-12-13T23:56:19.006267+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:21.936489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.002281+0000","last_change":"2023-12-13T23:56:18.998559+0000","last_active":"2023-12-13T23:56:19.002281+0000","last_peered":"2023-12-13T23:56:19.002281+0000","last_clean":"2023-12-13T23:56:19.002281+0000","last_became_active":"2023-12-13T23:56:18.998281+0000","last_became_peered":"2023-12-13T23:56:18.998281+0000","last_unstale":"2023-12-13T23:56:19.002281+0000","last_undegraded":"2023-12-13T23:56:19.002281+0000","last_fullsized":"2023-12-13T23:56:19.002281+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:30.765049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993777+0000","last_change":"2023-12-13T23:56:19.004339+0000","last_active":"2023-12-13T23:56:19.993777+0000","last_peered":"2023-12-13T23:56:19.993777+0000","last_clean":"2023-12-13T23:56:19.993777+0000","last_became_active":"2023-12-13T23:56:19.004017+0000","last_became_peered":"2023-12-13T23:56:19.004017+0000","last_unstale":"2023-12-13T23:56:19.993777+0000","last_undegraded":"2023-12-13T23:56:19.993777+0000","last_fullsized":"2023-12-13T23:56:19.993777+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:07:28.143709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993803+0000","last_change":"2023-12-13T23:56:18.999020+0000","last_active":"2023-12-13T23:56:19.993803+0000","last_peered":"2023-12-13T23:56:19.993803+0000","last_clean":"2023-12-13T23:56:19.993803+0000","last_became_active":"2023-12-13T23:56:18.998712+0000","last_became_peered":"2023-12-13T23:56:18.998712+0000","last_unstale":"2023-12-13T23:56:19.993803+0000","last_undegraded":"2023-12-13T23:56:19.993803+0000","last_fullsized":"2023-12-13T23:56:19.993803+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:24:32.727328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006274+0000","last_change":"2023-12-13T23:56:19.003820+0000","last_active":"2023-12-13T23:56:19.006274+0000","last_peered":"2023-12-13T23:56:19.006274+0000","last_clean":"2023-12-13T23:56:19.006274+0000","last_became_active":"2023-12-13T23:56:19.003561+0000","last_became_peered":"2023-12-13T23:56:19.003561+0000","last_unstale":"2023-12-13T23:56:19.006274+0000","last_undegraded":"2023-12-13T23:56:19.006274+0000","last_fullsized":"2023-12-13T23:56:19.006274+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:17:58.669960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006902+0000","last_change":"2023-12-13T23:56:19.003785+0000","last_active":"2023-12-13T23:56:19.006902+0000","last_peered":"2023-12-13T23:56:19.006902+0000","last_clean":"2023-12-13T23:56:19.006902+0000","last_became_active":"2023-12-13T23:56:19.003519+0000","last_became_peered":"2023-12-13T23:56:19.003519+0000","last_unstale":"2023-12-13T23:56:19.006902+0000","last_undegraded":"2023-12-13T23:56:19.006902+0000","last_fullsized":"2023-12-13T23:56:19.006902+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:08:26.045928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.988982+0000","last_change":"2023-12-13T23:56:19.001893+0000","last_active":"2023-12-13T23:56:19.988982+0000","last_peered":"2023-12-13T23:56:19.988982+0000","last_clean":"2023-12-13T23:56:19.988982+0000","last_became_active":"2023-12-13T23:56:19.001634+0000","last_became_peered":"2023-12-13T23:56:19.001634+0000","last_unstale":"2023-12-13T23:56:19.988982+0000","last_undegraded":"2023-12-13T23:56:19.988982+0000","last_fullsized":"2023-12-13T23:56:19.988982+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:25:37.104282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992466+0000","last_change":"2023-12-13T23:56:19.008649+0000","last_active":"2023-12-13T23:56:19.992466+0000","last_peered":"2023-12-13T23:56:19.992466+0000","last_clean":"2023-12-13T23:56:19.992466+0000","last_became_active":"2023-12-13T23:56:19.006114+0000","last_became_peered":"2023-12-13T23:56:19.006114+0000","last_unstale":"2023-12-13T23:56:19.992466+0000","last_undegraded":"2023-12-13T23:56:19.992466+0000","last_fullsized":"2023-12-13T23:56:19.992466+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:15:31.032348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992506+0000","last_change":"2023-12-13T23:56:19.006147+0000","last_active":"2023-12-13T23:56:19.992506+0000","last_peered":"2023-12-13T23:56:19.992506+0000","last_clean":"2023-12-13T23:56:19.992506+0000","last_became_active":"2023-12-13T23:56:19.005834+0000","last_became_peered":"2023-12-13T23:56:19.005834+0000","last_unstale":"2023-12-13T23:56:19.992506+0000","last_undegraded":"2023-12-13T23:56:19.992506+0000","last_fullsized":"2023-12-13T23:56:19.992506+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:08:26.045928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.004761+0000","last_change":"2023-12-13T23:56:19.003956+0000","last_active":"2023-12-13T23:56:19.004761+0000","last_peered":"2023-12-13T23:56:19.004761+0000","last_clean":"2023-12-13T23:56:19.004761+0000","last_became_active":"2023-12-13T23:56:19.003712+0000","last_became_peered":"2023-12-13T23:56:19.003712+0000","last_unstale":"2023-12-13T23:56:19.004761+0000","last_undegraded":"2023-12-13T23:56:19.004761+0000","last_fullsized":"2023-12-13T23:56:19.004761+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:55.196668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.997604+0000","last_change":"2023-12-13T23:56:18.995814+0000","last_active":"2023-12-13T23:56:18.997604+0000","last_peered":"2023-12-13T23:56:18.997604+0000","last_clean":"2023-12-13T23:56:18.997604+0000","last_became_active":"2023-12-13T23:56:18.995527+0000","last_became_peered":"2023-12-13T23:56:18.995527+0000","last_unstale":"2023-12-13T23:56:18.997604+0000","last_undegraded":"2023-12-13T23:56:18.997604+0000","last_fullsized":"2023-12-13T23:56:18.997604+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:16.467667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006910+0000","last_change":"2023-12-13T23:56:19.004807+0000","last_active":"2023-12-13T23:56:19.006910+0000","last_peered":"2023-12-13T23:56:19.006910+0000","last_clean":"2023-12-13T23:56:19.006910+0000","last_became_active":"2023-12-13T23:56:19.004568+0000","last_became_peered":"2023-12-13T23:56:19.004568+0000","last_unstale":"2023-12-13T23:56:19.006910+0000","last_undegraded":"2023-12-13T23:56:19.006910+0000","last_fullsized":"2023-12-13T23:56:19.006910+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:08:26.045928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013886+0000","last_change":"2023-12-13T23:56:19.011239+0000","last_active":"2023-12-13T23:56:19.013886+0000","last_peered":"2023-12-13T23:56:19.013886+0000","last_clean":"2023-12-13T23:56:19.013886+0000","last_became_active":"2023-12-13T23:56:19.010978+0000","last_became_peered":"2023-12-13T23:56:19.010978+0000","last_unstale":"2023-12-13T23:56:19.013886+0000","last_undegraded":"2023-12-13T23:56:19.013886+0000","last_fullsized":"2023-12-13T23:56:19.013886+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:29:26.833083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013805+0000","last_change":"2023-12-13T23:56:19.009883+0000","last_active":"2023-12-13T23:56:19.013805+0000","last_peered":"2023-12-13T23:56:19.013805+0000","last_clean":"2023-12-13T23:56:19.013805+0000","last_became_active":"2023-12-13T23:56:19.009640+0000","last_became_peered":"2023-12-13T23:56:19.009640+0000","last_unstale":"2023-12-13T23:56:19.013805+0000","last_undegraded":"2023-12-13T23:56:19.013805+0000","last_fullsized":"2023-12-13T23:56:19.013805+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:18:22.706473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007508+0000","last_change":"2023-12-13T23:56:19.002249+0000","last_active":"2023-12-13T23:56:19.007508+0000","last_peered":"2023-12-13T23:56:19.007508+0000","last_clean":"2023-12-13T23:56:19.007508+0000","last_became_active":"2023-12-13T23:56:19.001952+0000","last_became_peered":"2023-12-13T23:56:19.001952+0000","last_unstale":"2023-12-13T23:56:19.007508+0000","last_undegraded":"2023-12-13T23:56:19.007508+0000","last_fullsized":"2023-12-13T23:56:19.007508+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:15:31.032348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992943+0000","last_change":"2023-12-13T23:56:19.001641+0000","last_active":"2023-12-13T23:56:19.992943+0000","last_peered":"2023-12-13T23:56:19.992943+0000","last_clean":"2023-12-13T23:56:19.992943+0000","last_became_active":"2023-12-13T23:56:19.001390+0000","last_became_peered":"2023-12-13T23:56:19.001390+0000","last_unstale":"2023-12-13T23:56:19.992943+0000","last_undegraded":"2023-12-13T23:56:19.992943+0000","last_fullsized":"2023-12-13T23:56:19.992943+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:39:54.581072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993257+0000","last_change":"2023-12-13T23:56:18.999499+0000","last_active":"2023-12-13T23:56:19.993257+0000","last_peered":"2023-12-13T23:56:19.993257+0000","last_clean":"2023-12-13T23:56:19.993257+0000","last_became_active":"2023-12-13T23:56:18.998668+0000","last_became_peered":"2023-12-13T23:56:18.998668+0000","last_unstale":"2023-12-13T23:56:19.993257+0000","last_undegraded":"2023-12-13T23:56:19.993257+0000","last_fullsized":"2023-12-13T23:56:19.993257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:29:26.833083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007995+0000","last_change":"2023-12-13T23:56:19.002092+0000","last_active":"2023-12-13T23:56:19.007995+0000","last_peered":"2023-12-13T23:56:19.007995+0000","last_clean":"2023-12-13T23:56:19.007995+0000","last_became_active":"2023-12-13T23:56:19.001783+0000","last_became_peered":"2023-12-13T23:56:19.001783+0000","last_unstale":"2023-12-13T23:56:19.007995+0000","last_undegraded":"2023-12-13T23:56:19.007995+0000","last_fullsized":"2023-12-13T23:56:19.007995+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:39:52.370803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993215+0000","last_change":"2023-12-13T23:56:19.009981+0000","last_active":"2023-12-13T23:56:19.993215+0000","last_peered":"2023-12-13T23:56:19.993215+0000","last_clean":"2023-12-13T23:56:19.993215+0000","last_became_active":"2023-12-13T23:56:19.009722+0000","last_became_peered":"2023-12-13T23:56:19.009722+0000","last_unstale":"2023-12-13T23:56:19.993215+0000","last_undegraded":"2023-12-13T23:56:19.993215+0000","last_fullsized":"2023-12-13T23:56:19.993215+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:59:36.293125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007784+0000","last_change":"2023-12-13T23:56:19.005988+0000","last_active":"2023-12-13T23:56:19.007784+0000","last_peered":"2023-12-13T23:56:19.007784+0000","last_clean":"2023-12-13T23:56:19.007784+0000","last_became_active":"2023-12-13T23:56:19.005737+0000","last_became_peered":"2023-12-13T23:56:19.005737+0000","last_unstale":"2023-12-13T23:56:19.007784+0000","last_undegraded":"2023-12-13T23:56:19.007784+0000","last_fullsized":"2023-12-13T23:56:19.007784+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:39.201454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":15,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.011863+0000","last_change":"2023-12-13T23:56:17.992065+0000","last_active":"2023-12-13T23:56:19.011863+0000","last_peered":"2023-12-13T23:56:19.011863+0000","last_clean":"2023-12-13T23:56:19.011863+0000","last_became_active":"2023-12-13T23:56:17.991802+0000","last_became_peered":"2023-12-13T23:56:17.991802+0000","last_unstale":"2023-12-13T23:56:19.011863+0000","last_undegraded":"2023-12-13T23:56:19.011863+0000","last_fullsized":"2023-12-13T23:56:19.011863+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:20.932255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.010962+0000","last_change":"2023-12-13T23:56:19.007575+0000","last_active":"2023-12-13T23:56:19.010962+0000","last_peered":"2023-12-13T23:56:19.010962+0000","last_clean":"2023-12-13T23:56:19.010962+0000","last_became_active":"2023-12-13T23:56:19.007203+0000","last_became_peered":"2023-12-13T23:56:19.007203+0000","last_unstale":"2023-12-13T23:56:19.010962+0000","last_undegraded":"2023-12-13T23:56:19.010962+0000","last_fullsized":"2023-12-13T23:56:19.010962+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:39.201454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.009169+0000","last_change":"2023-12-13T23:56:17.984701+0000","last_active":"2023-12-13T23:56:19.009169+0000","last_peered":"2023-12-13T23:56:19.009169+0000","last_clean":"2023-12-13T23:56:19.009169+0000","last_became_active":"2023-12-13T23:56:17.984452+0000","last_became_peered":"2023-12-13T23:56:17.984452+0000","last_unstale":"2023-12-13T23:56:19.009169+0000","last_undegraded":"2023-12-13T23:56:19.009169+0000","last_fullsized":"2023-12-13T23:56:19.009169+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:20.932255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993425+0000","last_change":"2023-12-13T23:56:19.011597+0000","last_active":"2023-12-13T23:56:19.993425+0000","last_peered":"2023-12-13T23:56:19.993425+0000","last_clean":"2023-12-13T23:56:19.993425+0000","last_became_active":"2023-12-13T23:56:19.011208+0000","last_became_peered":"2023-12-13T23:56:19.011208+0000","last_unstale":"2023-12-13T23:56:19.993425+0000","last_undegraded":"2023-12-13T23:56:19.993425+0000","last_fullsized":"2023-12-13T23:56:19.993425+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:58.722565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.016006+0000","last_change":"2023-12-13T23:56:17.989802+0000","last_active":"2023-12-13T23:56:19.016006+0000","last_peered":"2023-12-13T23:56:19.016006+0000","last_clean":"2023-12-13T23:56:19.016006+0000","last_became_active":"2023-12-13T23:56:17.989526+0000","last_became_peered":"2023-12-13T23:56:17.989526+0000","last_unstale":"2023-12-13T23:56:19.016006+0000","last_undegraded":"2023-12-13T23:56:19.016006+0000","last_fullsized":"2023-12-13T23:56:19.016006+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:05:57.718330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013786+0000","last_change":"2023-12-13T23:56:19.009186+0000","last_active":"2023-12-13T23:56:19.013786+0000","last_peered":"2023-12-13T23:56:19.013786+0000","last_clean":"2023-12-13T23:56:19.013786+0000","last_became_active":"2023-12-13T23:56:19.008647+0000","last_became_peered":"2023-12-13T23:56:19.008647+0000","last_unstale":"2023-12-13T23:56:19.013786+0000","last_undegraded":"2023-12-13T23:56:19.013786+0000","last_fullsized":"2023-12-13T23:56:19.013786+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:30.765049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.989010+0000","last_change":"2023-12-13T23:56:17.986062+0000","last_active":"2023-12-13T23:56:19.989010+0000","last_peered":"2023-12-13T23:56:19.989010+0000","last_clean":"2023-12-13T23:56:19.989010+0000","last_became_active":"2023-12-13T23:56:17.985798+0000","last_became_peered":"2023-12-13T23:56:17.985798+0000","last_unstale":"2023-12-13T23:56:19.989010+0000","last_undegraded":"2023-12-13T23:56:19.989010+0000","last_fullsized":"2023-12-13T23:56:19.989010+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:09:24.462304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993685+0000","last_change":"2023-12-13T23:56:19.010582+0000","last_active":"2023-12-13T23:56:19.993685+0000","last_peered":"2023-12-13T23:56:19.993685+0000","last_clean":"2023-12-13T23:56:19.993685+0000","last_became_active":"2023-12-13T23:56:19.010250+0000","last_became_peered":"2023-12-13T23:56:19.010250+0000","last_unstale":"2023-12-13T23:56:19.993685+0000","last_undegraded":"2023-12-13T23:56:19.993685+0000","last_fullsized":"2023-12-13T23:56:19.993685+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:11:25.231381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992700+0000","last_change":"2023-12-13T23:56:17.993548+0000","last_active":"2023-12-13T23:56:19.992700+0000","last_peered":"2023-12-13T23:56:19.992700+0000","last_clean":"2023-12-13T23:56:19.992700+0000","last_became_active":"2023-12-13T23:56:17.992478+0000","last_became_peered":"2023-12-13T23:56:17.992478+0000","last_unstale":"2023-12-13T23:56:19.992700+0000","last_undegraded":"2023-12-13T23:56:19.992700+0000","last_fullsized":"2023-12-13T23:56:19.992700+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:15.463433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.010762+0000","last_change":"2023-12-13T23:56:19.003056+0000","last_active":"2023-12-13T23:56:19.010762+0000","last_peered":"2023-12-13T23:56:19.010762+0000","last_clean":"2023-12-13T23:56:19.010762+0000","last_became_active":"2023-12-13T23:56:19.002727+0000","last_became_peered":"2023-12-13T23:56:19.002727+0000","last_unstale":"2023-12-13T23:56:19.010762+0000","last_undegraded":"2023-12-13T23:56:19.010762+0000","last_fullsized":"2023-12-13T23:56:19.010762+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:39:54.581072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992644+0000","last_change":"2023-12-13T23:56:17.992714+0000","last_active":"2023-12-13T23:56:19.992644+0000","last_peered":"2023-12-13T23:56:19.992644+0000","last_clean":"2023-12-13T23:56:19.992644+0000","last_became_active":"2023-12-13T23:56:17.992288+0000","last_became_peered":"2023-12-13T23:56:17.992288+0000","last_unstale":"2023-12-13T23:56:19.992644+0000","last_undegraded":"2023-12-13T23:56:19.992644+0000","last_fullsized":"2023-12-13T23:56:19.992644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:38.197220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992361+0000","last_change":"2023-12-13T23:56:18.998439+0000","last_active":"2023-12-13T23:56:19.992361+0000","last_peered":"2023-12-13T23:56:19.992361+0000","last_clean":"2023-12-13T23:56:19.992361+0000","last_became_active":"2023-12-13T23:56:18.998168+0000","last_became_peered":"2023-12-13T23:56:18.998168+0000","last_unstale":"2023-12-13T23:56:19.992361+0000","last_undegraded":"2023-12-13T23:56:19.992361+0000","last_fullsized":"2023-12-13T23:56:19.992361+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:07:16.341679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.992917+0000","last_change":"2023-12-13T23:56:17.990191+0000","last_active":"2023-12-13T23:56:18.992917+0000","last_peered":"2023-12-13T23:56:18.992917+0000","last_clean":"2023-12-13T23:56:18.992917+0000","last_became_active":"2023-12-13T23:56:17.989910+0000","last_became_peered":"2023-12-13T23:56:17.989910+0000","last_unstale":"2023-12-13T23:56:18.992917+0000","last_undegraded":"2023-12-13T23:56:18.992917+0000","last_fullsized":"2023-12-13T23:56:18.992917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:54.748475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.002579+0000","last_change":"2023-12-13T23:56:18.999821+0000","last_active":"2023-12-13T23:56:19.002579+0000","last_peered":"2023-12-13T23:56:19.002579+0000","last_clean":"2023-12-13T23:56:19.002579+0000","last_became_active":"2023-12-13T23:56:18.999565+0000","last_became_peered":"2023-12-13T23:56:18.999565+0000","last_unstale":"2023-12-13T23:56:19.002579+0000","last_undegraded":"2023-12-13T23:56:19.002579+0000","last_fullsized":"2023-12-13T23:56:19.002579+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:25:01.043631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992468+0000","last_change":"2023-12-13T23:56:17.997038+0000","last_active":"2023-12-13T23:56:19.992468+0000","last_peered":"2023-12-13T23:56:19.992468+0000","last_clean":"2023-12-13T23:56:19.992468+0000","last_became_active":"2023-12-13T23:56:17.996755+0000","last_became_peered":"2023-12-13T23:56:17.996755+0000","last_unstale":"2023-12-13T23:56:19.992468+0000","last_undegraded":"2023-12-13T23:56:19.992468+0000","last_fullsized":"2023-12-13T23:56:19.992468+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-14T23:57:31.262514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007392+0000","last_change":"2023-12-13T23:56:19.002041+0000","last_active":"2023-12-13T23:56:19.007392+0000","last_peered":"2023-12-13T23:56:19.007392+0000","last_clean":"2023-12-13T23:56:19.007392+0000","last_became_active":"2023-12-13T23:56:19.001749+0000","last_became_peered":"2023-12-13T23:56:19.001749+0000","last_unstale":"2023-12-13T23:56:19.007392+0000","last_undegraded":"2023-12-13T23:56:19.007392+0000","last_fullsized":"2023-12-13T23:56:19.007392+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:07:16.341679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.991288+0000","last_change":"2023-12-13T23:56:17.989047+0000","last_active":"2023-12-13T23:56:18.991288+0000","last_peered":"2023-12-13T23:56:18.991288+0000","last_clean":"2023-12-13T23:56:18.991288+0000","last_became_active":"2023-12-13T23:56:17.988611+0000","last_became_peered":"2023-12-13T23:56:17.988611+0000","last_unstale":"2023-12-13T23:56:18.991288+0000","last_undegraded":"2023-12-13T23:56:18.991288+0000","last_fullsized":"2023-12-13T23:56:18.991288+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:29.760814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013464+0000","last_change":"2023-12-13T23:56:19.007029+0000","last_active":"2023-12-13T23:56:19.013464+0000","last_peered":"2023-12-13T23:56:19.013464+0000","last_clean":"2023-12-13T23:56:19.013464+0000","last_became_active":"2023-12-13T23:56:19.006689+0000","last_became_peered":"2023-12-13T23:56:19.006689+0000","last_unstale":"2023-12-13T23:56:19.013464+0000","last_undegraded":"2023-12-13T23:56:19.013464+0000","last_fullsized":"2023-12-13T23:56:19.013464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:12:15.168346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.997947+0000","last_change":"2023-12-13T23:56:17.991227+0000","last_active":"2023-12-13T23:56:18.997947+0000","last_peered":"2023-12-13T23:56:18.997947+0000","last_clean":"2023-12-13T23:56:18.997947+0000","last_became_active":"2023-12-13T23:56:17.990949+0000","last_became_peered":"2023-12-13T23:56:17.990949+0000","last_unstale":"2023-12-13T23:56:18.997947+0000","last_undegraded":"2023-12-13T23:56:18.997947+0000","last_fullsized":"2023-12-13T23:56:18.997947+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:12:14.164111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.004021+0000","last_change":"2023-12-13T23:56:19.001212+0000","last_active":"2023-12-13T23:56:19.004021+0000","last_peered":"2023-12-13T23:56:19.004021+0000","last_clean":"2023-12-13T23:56:19.004021+0000","last_became_active":"2023-12-13T23:56:19.000945+0000","last_became_peered":"2023-12-13T23:56:19.000945+0000","last_unstale":"2023-12-13T23:56:19.004021+0000","last_undegraded":"2023-12-13T23:56:19.004021+0000","last_fullsized":"2023-12-13T23:56:19.004021+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:16.467667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":14,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.018576+0000","last_change":"2023-12-13T23:56:17.990318+0000","last_active":"2023-12-13T23:56:19.018576+0000","last_peered":"2023-12-13T23:56:19.018576+0000","last_clean":"2023-12-13T23:56:19.018576+0000","last_became_active":"2023-12-13T23:56:17.989870+0000","last_became_peered":"2023-12-13T23:56:17.989870+0000","last_unstale":"2023-12-13T23:56:19.018576+0000","last_undegraded":"2023-12-13T23:56:19.018576+0000","last_fullsized":"2023-12-13T23:56:19.018576+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:27.722378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.010962+0000","last_change":"2023-12-13T23:56:19.004519+0000","last_active":"2023-12-13T23:56:19.010962+0000","last_peered":"2023-12-13T23:56:19.010962+0000","last_clean":"2023-12-13T23:56:19.010962+0000","last_became_active":"2023-12-13T23:56:19.004251+0000","last_became_peered":"2023-12-13T23:56:19.004251+0000","last_unstale":"2023-12-13T23:56:19.010962+0000","last_undegraded":"2023-12-13T23:56:19.010962+0000","last_fullsized":"2023-12-13T23:56:19.010962+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:18:22.706473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.020950+0000","last_change":"2023-12-13T23:56:17.991687+0000","last_active":"2023-12-13T23:56:19.020950+0000","last_peered":"2023-12-13T23:56:19.020950+0000","last_clean":"2023-12-13T23:56:19.020950+0000","last_became_active":"2023-12-13T23:56:17.991453+0000","last_became_peered":"2023-12-13T23:56:17.991453+0000","last_unstale":"2023-12-13T23:56:19.020950+0000","last_undegraded":"2023-12-13T23:56:19.020950+0000","last_fullsized":"2023-12-13T23:56:19.020950+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:39:53.576837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.002873+0000","last_change":"2023-12-13T23:56:19.001856+0000","last_active":"2023-12-13T23:56:19.002873+0000","last_peered":"2023-12-13T23:56:19.002873+0000","last_clean":"2023-12-13T23:56:19.002873+0000","last_became_active":"2023-12-13T23:56:19.001534+0000","last_became_peered":"2023-12-13T23:56:19.001534+0000","last_unstale":"2023-12-13T23:56:19.002873+0000","last_undegraded":"2023-12-13T23:56:19.002873+0000","last_fullsized":"2023-12-13T23:56:19.002873+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:58:39.201454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993045+0000","last_change":"2023-12-13T23:56:17.992938+0000","last_active":"2023-12-13T23:56:19.993045+0000","last_peered":"2023-12-13T23:56:19.993045+0000","last_clean":"2023-12-13T23:56:19.993045+0000","last_became_active":"2023-12-13T23:56:17.992632+0000","last_became_peered":"2023-12-13T23:56:17.992632+0000","last_unstale":"2023-12-13T23:56:19.993045+0000","last_undegraded":"2023-12-13T23:56:19.993045+0000","last_fullsized":"2023-12-13T23:56:19.993045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:29.760814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013839+0000","last_change":"2023-12-13T23:56:19.010939+0000","last_active":"2023-12-13T23:56:19.013839+0000","last_peered":"2023-12-13T23:56:19.013839+0000","last_clean":"2023-12-13T23:56:19.013839+0000","last_became_active":"2023-12-13T23:56:19.010573+0000","last_became_peered":"2023-12-13T23:56:19.010573+0000","last_unstale":"2023-12-13T23:56:19.013839+0000","last_undegraded":"2023-12-13T23:56:19.013839+0000","last_fullsized":"2023-12-13T23:56:19.013839+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:57:12.686117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.993325+0000","last_change":"2023-12-13T23:56:17.992624+0000","last_active":"2023-12-13T23:56:18.993325+0000","last_peered":"2023-12-13T23:56:18.993325+0000","last_clean":"2023-12-13T23:56:18.993325+0000","last_became_active":"2023-12-13T23:56:17.992287+0000","last_became_peered":"2023-12-13T23:56:17.992287+0000","last_unstale":"2023-12-13T23:56:18.993325+0000","last_undegraded":"2023-12-13T23:56:18.993325+0000","last_fullsized":"2023-12-13T23:56:18.993325+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:07:15.337444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993481+0000","last_change":"2023-12-13T23:56:19.009375+0000","last_active":"2023-12-13T23:56:19.993481+0000","last_peered":"2023-12-13T23:56:19.993481+0000","last_clean":"2023-12-13T23:56:19.993481+0000","last_became_active":"2023-12-13T23:56:19.009108+0000","last_became_peered":"2023-12-13T23:56:19.009108+0000","last_unstale":"2023-12-13T23:56:19.993481+0000","last_undegraded":"2023-12-13T23:56:19.993481+0000","last_fullsized":"2023-12-13T23:56:19.993481+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:17:58.669960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.993039+0000","last_change":"2023-12-13T23:56:17.990050+0000","last_active":"2023-12-13T23:56:18.993039+0000","last_peered":"2023-12-13T23:56:18.993039+0000","last_clean":"2023-12-13T23:56:18.993039+0000","last_became_active":"2023-12-13T23:56:17.989599+0000","last_became_peered":"2023-12-13T23:56:17.989599+0000","last_unstale":"2023-12-13T23:56:18.993039+0000","last_undegraded":"2023-12-13T23:56:18.993039+0000","last_fullsized":"2023-12-13T23:56:18.993039+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:27.722378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.010933+0000","last_change":"2023-12-13T23:56:19.006769+0000","last_active":"2023-12-13T23:56:19.010933+0000","last_peered":"2023-12-13T23:56:19.010933+0000","last_clean":"2023-12-13T23:56:19.010933+0000","last_became_active":"2023-12-13T23:56:19.006467+0000","last_became_peered":"2023-12-13T23:56:19.006467+0000","last_unstale":"2023-12-13T23:56:19.010933+0000","last_undegraded":"2023-12-13T23:56:19.010933+0000","last_fullsized":"2023-12-13T23:56:19.010933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:41:06.823180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.015936+0000","last_change":"2023-12-13T23:56:17.989563+0000","last_active":"2023-12-13T23:56:19.015936+0000","last_peered":"2023-12-13T23:56:19.015936+0000","last_clean":"2023-12-13T23:56:19.015936+0000","last_became_active":"2023-12-13T23:56:17.989329+0000","last_became_peered":"2023-12-13T23:56:17.989329+0000","last_unstale":"2023-12-13T23:56:19.015936+0000","last_undegraded":"2023-12-13T23:56:19.015936+0000","last_fullsized":"2023-12-13T23:56:19.015936+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:15:30.028113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993157+0000","last_change":"2023-12-13T23:56:19.009198+0000","last_active":"2023-12-13T23:56:19.993157+0000","last_peered":"2023-12-13T23:56:19.993157+0000","last_clean":"2023-12-13T23:56:19.993157+0000","last_became_active":"2023-12-13T23:56:19.008877+0000","last_became_peered":"2023-12-13T23:56:19.008877+0000","last_unstale":"2023-12-13T23:56:19.993157+0000","last_undegraded":"2023-12-13T23:56:19.993157+0000","last_fullsized":"2023-12-13T23:56:19.993157+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:57:12.686117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007360+0000","last_change":"2023-12-13T23:56:17.991584+0000","last_active":"2023-12-13T23:56:19.007360+0000","last_peered":"2023-12-13T23:56:19.007360+0000","last_clean":"2023-12-13T23:56:19.007360+0000","last_became_active":"2023-12-13T23:56:17.991290+0000","last_became_peered":"2023-12-13T23:56:17.991290+0000","last_unstale":"2023-12-13T23:56:19.007360+0000","last_undegraded":"2023-12-13T23:56:19.007360+0000","last_fullsized":"2023-12-13T23:56:19.007360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:39:53.576837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.014436+0000","last_change":"2023-12-13T23:56:19.008109+0000","last_active":"2023-12-13T23:56:19.014436+0000","last_peered":"2023-12-13T23:56:19.014436+0000","last_clean":"2023-12-13T23:56:19.014436+0000","last_became_active":"2023-12-13T23:56:19.007779+0000","last_became_peered":"2023-12-13T23:56:19.007779+0000","last_unstale":"2023-12-13T23:56:19.014436+0000","last_undegraded":"2023-12-13T23:56:19.014436+0000","last_fullsized":"2023-12-13T23:56:19.014436+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:57:12.686117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993246+0000","last_change":"2023-12-13T23:56:17.992825+0000","last_active":"2023-12-13T23:56:19.993246+0000","last_peered":"2023-12-13T23:56:19.993246+0000","last_clean":"2023-12-13T23:56:19.993246+0000","last_became_active":"2023-12-13T23:56:17.992527+0000","last_became_peered":"2023-12-13T23:56:17.992527+0000","last_unstale":"2023-12-13T23:56:19.993246+0000","last_undegraded":"2023-12-13T23:56:19.993246+0000","last_fullsized":"2023-12-13T23:56:19.993246+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:54.192433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.013872+0000","last_change":"2023-12-13T23:56:19.011326+0000","last_active":"2023-12-13T23:56:19.013872+0000","last_peered":"2023-12-13T23:56:19.013872+0000","last_clean":"2023-12-13T23:56:19.013872+0000","last_became_active":"2023-12-13T23:56:19.010497+0000","last_became_peered":"2023-12-13T23:56:19.010497+0000","last_unstale":"2023-12-13T23:56:19.013872+0000","last_undegraded":"2023-12-13T23:56:19.013872+0000","last_fullsized":"2023-12-13T23:56:19.013872+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:21.936489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993073+0000","last_change":"2023-12-13T23:56:17.993522+0000","last_active":"2023-12-13T23:56:19.993073+0000","last_peered":"2023-12-13T23:56:19.993073+0000","last_clean":"2023-12-13T23:56:19.993073+0000","last_became_active":"2023-12-13T23:56:17.993228+0000","last_became_peered":"2023-12-13T23:56:17.993228+0000","last_unstale":"2023-12-13T23:56:19.993073+0000","last_undegraded":"2023-12-13T23:56:19.993073+0000","last_fullsized":"2023-12-13T23:56:19.993073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:25:36.100047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.989149+0000","last_change":"2023-12-13T23:56:18.999511+0000","last_active":"2023-12-13T23:56:19.989149+0000","last_peered":"2023-12-13T23:56:19.989149+0000","last_clean":"2023-12-13T23:56:19.989149+0000","last_became_active":"2023-12-13T23:56:18.999248+0000","last_became_peered":"2023-12-13T23:56:18.999248+0000","last_unstale":"2023-12-13T23:56:19.989149+0000","last_undegraded":"2023-12-13T23:56:19.989149+0000","last_fullsized":"2023-12-13T23:56:19.989149+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:55.196668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992880+0000","last_change":"2023-12-13T23:56:17.992764+0000","last_active":"2023-12-13T23:56:19.992880+0000","last_peered":"2023-12-13T23:56:19.992880+0000","last_clean":"2023-12-13T23:56:19.992880+0000","last_became_active":"2023-12-13T23:56:17.992485+0000","last_became_peered":"2023-12-13T23:56:17.992485+0000","last_unstale":"2023-12-13T23:56:19.992880+0000","last_undegraded":"2023-12-13T23:56:19.992880+0000","last_fullsized":"2023-12-13T23:56:19.992880+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:22:20.932255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.988927+0000","last_change":"2023-12-13T23:56:19.007397+0000","last_active":"2023-12-13T23:56:19.988927+0000","last_peered":"2023-12-13T23:56:19.988927+0000","last_clean":"2023-12-13T23:56:19.988927+0000","last_became_active":"2023-12-13T23:56:19.007187+0000","last_became_peered":"2023-12-13T23:56:19.007187+0000","last_unstale":"2023-12-13T23:56:19.988927+0000","last_undegraded":"2023-12-13T23:56:19.988927+0000","last_fullsized":"2023-12-13T23:56:19.988927+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-14T23:57:32.266749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.009717+0000","last_change":"2023-12-13T23:56:17.984295+0000","last_active":"2023-12-13T23:56:19.009717+0000","last_peered":"2023-12-13T23:56:19.009717+0000","last_clean":"2023-12-13T23:56:19.009717+0000","last_became_active":"2023-12-13T23:56:17.983873+0000","last_became_peered":"2023-12-13T23:56:17.983873+0000","last_unstale":"2023-12-13T23:56:19.009717+0000","last_undegraded":"2023-12-13T23:56:19.009717+0000","last_fullsized":"2023-12-13T23:56:19.009717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:30:29.760814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.989079+0000","last_change":"2023-12-13T23:56:19.002474+0000","last_active":"2023-12-13T23:56:19.989079+0000","last_peered":"2023-12-13T23:56:19.989079+0000","last_clean":"2023-12-13T23:56:19.989079+0000","last_became_active":"2023-12-13T23:56:19.002215+0000","last_became_peered":"2023-12-13T23:56:19.002215+0000","last_unstale":"2023-12-13T23:56:19.989079+0000","last_undegraded":"2023-12-13T23:56:19.989079+0000","last_fullsized":"2023-12-13T23:56:19.989079+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:39:52.370803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.993850+0000","last_change":"2023-12-13T23:56:17.990714+0000","last_active":"2023-12-13T23:56:18.993850+0000","last_peered":"2023-12-13T23:56:18.993850+0000","last_clean":"2023-12-13T23:56:18.993850+0000","last_became_active":"2023-12-13T23:56:17.990380+0000","last_became_peered":"2023-12-13T23:56:17.990380+0000","last_unstale":"2023-12-13T23:56:18.993850+0000","last_undegraded":"2023-12-13T23:56:18.993850+0000","last_fullsized":"2023-12-13T23:56:18.993850+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:25:00.039396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.988887+0000","last_change":"2023-12-13T23:56:19.001852+0000","last_active":"2023-12-13T23:56:19.988887+0000","last_peered":"2023-12-13T23:56:19.988887+0000","last_clean":"2023-12-13T23:56:19.988887+0000","last_became_active":"2023-12-13T23:56:19.001585+0000","last_became_peered":"2023-12-13T23:56:19.001585+0000","last_unstale":"2023-12-13T23:56:19.988887+0000","last_undegraded":"2023-12-13T23:56:19.988887+0000","last_fullsized":"2023-12-13T23:56:19.988887+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:24:24.457323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992873+0000","last_change":"2023-12-13T23:56:17.996506+0000","last_active":"2023-12-13T23:56:19.992873+0000","last_peered":"2023-12-13T23:56:19.992873+0000","last_clean":"2023-12-13T23:56:19.992873+0000","last_became_active":"2023-12-13T23:56:17.996184+0000","last_became_peered":"2023-12-13T23:56:17.996184+0000","last_unstale":"2023-12-13T23:56:19.992873+0000","last_undegraded":"2023-12-13T23:56:19.992873+0000","last_fullsized":"2023-12-13T23:56:19.992873+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:37.013270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992719+0000","last_change":"2023-12-13T23:56:18.998532+0000","last_active":"2023-12-13T23:56:19.992719+0000","last_peered":"2023-12-13T23:56:19.992719+0000","last_clean":"2023-12-13T23:56:19.992719+0000","last_became_active":"2023-12-13T23:56:18.998279+0000","last_became_peered":"2023-12-13T23:56:18.998279+0000","last_unstale":"2023-12-13T23:56:19.992719+0000","last_undegraded":"2023-12-13T23:56:19.992719+0000","last_fullsized":"2023-12-13T23:56:19.992719+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:28.726612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.989080+0000","last_change":"2023-12-13T23:56:17.985244+0000","last_active":"2023-12-13T23:56:18.989080+0000","last_peered":"2023-12-13T23:56:18.989080+0000","last_clean":"2023-12-13T23:56:18.989080+0000","last_became_active":"2023-12-13T23:56:17.984960+0000","last_became_peered":"2023-12-13T23:56:17.984960+0000","last_unstale":"2023-12-13T23:56:18.989080+0000","last_undegraded":"2023-12-13T23:56:18.989080+0000","last_fullsized":"2023-12-13T23:56:18.989080+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:12:14.164111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.988992+0000","last_change":"2023-12-13T23:56:19.002267+0000","last_active":"2023-12-13T23:56:19.988992+0000","last_peered":"2023-12-13T23:56:19.988992+0000","last_clean":"2023-12-13T23:56:19.988992+0000","last_became_active":"2023-12-13T23:56:19.001985+0000","last_became_peered":"2023-12-13T23:56:19.001985+0000","last_unstale":"2023-12-13T23:56:19.988992+0000","last_undegraded":"2023-12-13T23:56:19.988992+0000","last_fullsized":"2023-12-13T23:56:19.988992+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:41:06.823180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.008385+0000","last_change":"2023-12-13T23:56:17.991661+0000","last_active":"2023-12-13T23:56:19.008385+0000","last_peered":"2023-12-13T23:56:19.008385+0000","last_clean":"2023-12-13T23:56:19.008385+0000","last_became_active":"2023-12-13T23:56:17.991408+0000","last_became_peered":"2023-12-13T23:56:17.991408+0000","last_unstale":"2023-12-13T23:56:19.008385+0000","last_undegraded":"2023-12-13T23:56:19.008385+0000","last_fullsized":"2023-12-13T23:56:19.008385+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:57:11.681882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.014411+0000","last_change":"2023-12-13T23:56:19.007756+0000","last_active":"2023-12-13T23:56:19.014411+0000","last_peered":"2023-12-13T23:56:19.014411+0000","last_clean":"2023-12-13T23:56:19.014411+0000","last_became_active":"2023-12-13T23:56:19.007321+0000","last_became_peered":"2023-12-13T23:56:19.007321+0000","last_unstale":"2023-12-13T23:56:19.014411+0000","last_undegraded":"2023-12-13T23:56:19.014411+0000","last_fullsized":"2023-12-13T23:56:19.014411+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:55.752709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.998069+0000","last_change":"2023-12-13T23:56:17.987634+0000","last_active":"2023-12-13T23:56:18.998069+0000","last_peered":"2023-12-13T23:56:18.998069+0000","last_clean":"2023-12-13T23:56:18.998069+0000","last_became_active":"2023-12-13T23:56:17.987398+0000","last_became_peered":"2023-12-13T23:56:17.987398+0000","last_unstale":"2023-12-13T23:56:18.998069+0000","last_undegraded":"2023-12-13T23:56:18.998069+0000","last_fullsized":"2023-12-13T23:56:18.998069+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:50:15.463433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.006566+0000","last_change":"2023-12-13T23:56:19.004882+0000","last_active":"2023-12-13T23:56:19.006566+0000","last_peered":"2023-12-13T23:56:19.006566+0000","last_clean":"2023-12-13T23:56:19.006566+0000","last_became_active":"2023-12-13T23:56:19.004623+0000","last_became_peered":"2023-12-13T23:56:19.004623+0000","last_unstale":"2023-12-13T23:56:19.006566+0000","last_undegraded":"2023-12-13T23:56:19.006566+0000","last_fullsized":"2023-12-13T23:56:19.006566+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:25:01.043631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.994209+0000","last_change":"2023-12-13T23:56:17.983989+0000","last_active":"2023-12-13T23:56:18.994209+0000","last_peered":"2023-12-13T23:56:18.994209+0000","last_clean":"2023-12-13T23:56:18.994209+0000","last_became_active":"2023-12-13T23:56:17.983734+0000","last_became_peered":"2023-12-13T23:56:17.983734+0000","last_unstale":"2023-12-13T23:56:18.994209+0000","last_undegraded":"2023-12-13T23:56:18.994209+0000","last_fullsized":"2023-12-13T23:56:18.994209+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:54.748475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.007968+0000","last_change":"2023-12-13T23:56:19.002687+0000","last_active":"2023-12-13T23:56:19.007968+0000","last_peered":"2023-12-13T23:56:19.007968+0000","last_clean":"2023-12-13T23:56:19.007968+0000","last_became_active":"2023-12-13T23:56:19.002434+0000","last_became_peered":"2023-12-13T23:56:19.002434+0000","last_unstale":"2023-12-13T23:56:19.007968+0000","last_undegraded":"2023-12-13T23:56:19.007968+0000","last_fullsized":"2023-12-13T23:56:19.007968+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:20:55.196668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.998429+0000","last_change":"2023-12-13T23:56:17.986688+0000","last_active":"2023-12-13T23:56:18.998429+0000","last_peered":"2023-12-13T23:56:18.998429+0000","last_clean":"2023-12-13T23:56:18.998429+0000","last_became_active":"2023-12-13T23:56:17.986318+0000","last_became_peered":"2023-12-13T23:56:17.986318+0000","last_unstale":"2023-12-13T23:56:18.998429+0000","last_undegraded":"2023-12-13T23:56:18.998429+0000","last_fullsized":"2023-12-13T23:56:18.998429+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:18:21.702239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.008926+0000","last_change":"2023-12-13T23:56:19.005583+0000","last_active":"2023-12-13T23:56:19.008926+0000","last_peered":"2023-12-13T23:56:19.008926+0000","last_clean":"2023-12-13T23:56:19.008926+0000","last_became_active":"2023-12-13T23:56:19.005242+0000","last_became_peered":"2023-12-13T23:56:19.005242+0000","last_unstale":"2023-12-13T23:56:19.008926+0000","last_undegraded":"2023-12-13T23:56:19.008926+0000","last_fullsized":"2023-12-13T23:56:19.008926+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:44:39.483806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.993448+0000","last_change":"2023-12-13T23:56:17.992066+0000","last_active":"2023-12-13T23:56:18.993448+0000","last_peered":"2023-12-13T23:56:18.993448+0000","last_clean":"2023-12-13T23:56:18.993448+0000","last_became_active":"2023-12-13T23:56:17.991823+0000","last_became_peered":"2023-12-13T23:56:17.991823+0000","last_unstale":"2023-12-13T23:56:18.993448+0000","last_undegraded":"2023-12-13T23:56:18.993448+0000","last_fullsized":"2023-12-13T23:56:18.993448+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:15:30.028113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.011816+0000","last_change":"2023-12-13T23:56:19.005236+0000","last_active":"2023-12-13T23:56:19.011816+0000","last_peered":"2023-12-13T23:56:19.011816+0000","last_clean":"2023-12-13T23:56:19.011816+0000","last_became_active":"2023-12-13T23:56:19.003938+0000","last_became_peered":"2023-12-13T23:56:19.003938+0000","last_unstale":"2023-12-13T23:56:19.011816+0000","last_undegraded":"2023-12-13T23:56:19.011816+0000","last_fullsized":"2023-12-13T23:56:19.011816+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:07:16.341679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992964+0000","last_change":"2023-12-13T23:56:17.994238+0000","last_active":"2023-12-13T23:56:19.992964+0000","last_peered":"2023-12-13T23:56:19.992964+0000","last_clean":"2023-12-13T23:56:19.992964+0000","last_became_active":"2023-12-13T23:56:17.993973+0000","last_became_peered":"2023-12-13T23:56:17.993973+0000","last_unstale":"2023-12-13T23:56:19.992964+0000","last_undegraded":"2023-12-13T23:56:19.992964+0000","last_fullsized":"2023-12-13T23:56:19.992964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:41:05.818945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.993372+0000","last_change":"2023-12-13T23:56:19.002051+0000","last_active":"2023-12-13T23:56:19.993372+0000","last_peered":"2023-12-13T23:56:19.993372+0000","last_clean":"2023-12-13T23:56:19.993372+0000","last_became_active":"2023-12-13T23:56:19.001761+0000","last_became_peered":"2023-12-13T23:56:19.001761+0000","last_unstale":"2023-12-13T23:56:19.993372+0000","last_undegraded":"2023-12-13T23:56:19.993372+0000","last_fullsized":"2023-12-13T23:56:19.993372+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:11:28.726612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.993762+0000","last_change":"2023-12-13T23:56:17.990306+0000","last_active":"2023-12-13T23:56:18.993762+0000","last_peered":"2023-12-13T23:56:18.993762+0000","last_clean":"2023-12-13T23:56:18.993762+0000","last_became_active":"2023-12-13T23:56:17.989745+0000","last_became_peered":"2023-12-13T23:56:17.989745+0000","last_unstale":"2023-12-13T23:56:18.993762+0000","last_undegraded":"2023-12-13T23:56:18.993762+0000","last_fullsized":"2023-12-13T23:56:18.993762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:59:35.288890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2023-12-13T23:56:19.992457+0000","last_change":"2023-12-13T23:56:19.006703+0000","last_active":"2023-12-13T23:56:19.992457+0000","last_peered":"2023-12-13T23:56:19.992457+0000","last_clean":"2023-12-13T23:56:19.992457+0000","last_became_active":"2023-12-13T23:56:19.006443+0000","last_became_peered":"2023-12-13T23:56:19.006443+0000","last_unstale":"2023-12-13T23:56:19.992457+0000","last_undegraded":"2023-12-13T23:56:19.992457+0000","last_fullsized":"2023-12-13T23:56:19.992457+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:17.970612+0000","last_clean_scrub_stamp":"2023-12-13T23:56:17.970612+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:57:55.752709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.992131+0000","last_change":"2023-12-13T23:56:17.996394+0000","last_active":"2023-12-13T23:56:18.992131+0000","last_peered":"2023-12-13T23:56:18.992131+0000","last_clean":"2023-12-13T23:56:18.992131+0000","last_became_active":"2023-12-13T23:56:17.996084+0000","last_became_peered":"2023-12-13T23:56:17.996084+0000","last_unstale":"2023-12-13T23:56:18.992131+0000","last_undegraded":"2023-12-13T23:56:18.992131+0000","last_fullsized":"2023-12-13T23:56:18.992131+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:56:16.966377+0000","last_clean_scrub_stamp":"2023-12-13T23:56:16.966377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:25:36.100047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":179,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-12-13T23:56:18.992267+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:56:18.992267+0000","last_peered":"2023-12-13T23:56:18.992267+0000","last_clean":"2023-12-13T23:56:18.992267+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:56:18.992267+0000","last_undegraded":"2023-12-13T23:56:18.992267+0000","last_fullsized":"2023-12-13T23:56:18.992267+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":747036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55800000000000005}]},{"osd":2,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.75},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.66000000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.194,"5min":1.194,"15min":1.194},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.73899999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72199999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574869,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":150894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52100000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":3.5190000000000001,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.45800000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47199999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.441}]},{"osd":7,"last update":"Wed Dec 13 23:56:04 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.48899999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":150950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.498},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.46400000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.51700000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.627},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47899999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.439},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.41599999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.60299999999999998}]},{"osd":7,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.65700000000000003}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":393216,"data_stored":150382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.47999999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55300000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:56:00 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64400000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":151912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"last":0.379},{"interface":"front","average":{"1min":0.316,"5min":0.316,"15min":0.316},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"last":0.26700000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"last":0.35599999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.317}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.56999999999999995}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.59999999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.59199999999999997}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27400,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344440,"statfs":{"total":96636764160,"available":96608706560,"internally_reserved":0,"allocated":389120,"data_stored":150348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.30299999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.376}]},{"osd":3,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.318}]},{"osd":4,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.62}]},{"osd":5,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.63}]},{"osd":6,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56399999999999995}]},{"osd":7,"last update":"Wed Dec 13 23:56:03 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55100000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27412,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344428,"statfs":{"total":96636764160,"available":96608694272,"internally_reserved":0,"allocated":401408,"data_stored":151912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.254},{"interface":"front","average":{"1min":0.378,"5min":0.378,"15min":0.378},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3520000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.27600000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.6619999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.29199999999999998}]},{"osd":3,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.512,"5min":1.512,"15min":1.512},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":1.401,"5min":1.401,"15min":1.401},"last":0.308}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.6379999999999999,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.59499999999999997}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.503},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.72899999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.292,"5min":1.292,"15min":1.292},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.51300000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.377,"5min":1.377,"15min":1.377},"last":0.57699999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":978944,"data_stored":740646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.316}]},{"osd":2,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.377}]},{"osd":3,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.36399999999999999}]},{"osd":4,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.64400000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.627,"15min":0.627},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.63}]},{"osd":6,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Wed Dec 13 23:56:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.454,"5min":1.454,"15min":1.454},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.56999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:56:22.260 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-13T23:56:22.261 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-13T23:56:22.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.403+0000 7f9926768640 1 Processor -- start 2023-12-13T23:56:22.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.403+0000 7f9926768640 1 -- start start 2023-12-13T23:56:22.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f9926768640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9920141930 0x7f99200ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f9926768640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9920141d10 con 0x7f992015f8c0 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f9926768640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992015fdc0 con 0x7f99200b1300 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f9926768640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99200cd320 con 0x7f9920141930 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f991f7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f992015f8c0 0x7f99200b0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36218/0 (socket says 172.21.15.124:36218) 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f991f7fe640 1 -- 172.21.15.124:0/1664910497 learned_addr learned my addr 172.21.15.124:0/1664910497 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f991ffff640 1 --2- 172.21.15.124:0/1664910497 >> v2:172.21.15.203:3300/0 conn(0x7f9920141930 0x7f99200ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698867920 0 0) 0x7f9920141d10 con 0x7f992015f8c0 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.404+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9908003520 con 0x7f992015f8c0 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927720545 0 0) 0x7f9908003520 con 0x7f992015f8c0 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 >> v2:172.21.15.203:3300/0 conn(0x7f9920141930 msgr2=0x7f99200ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 --2- 172.21.15.124:0/1664910497 >> v2:172.21.15.203:3300/0 conn(0x7f9920141930 0x7f99200ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 >> v1:172.21.15.203:6789/0 conn(0x7f99200b1300 legacy=0x7f9920141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99200cd600 con 0x7f992015f8c0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f9914001690 con 0x7f992015f8c0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 >> v1:172.21.15.124:6789/0 conn(0x7f992015f8c0 legacy=0x7f99200b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9908003970 0x7f9908003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 0x7f9908004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9908005180 0x7f990800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99200cd320 con 0x7f99080047c0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990800c120 con 0x7f9908003970 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990800c2e0 con 0x7f9908005180 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 0x7f9908004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991ffff640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9908003970 0x7f9908003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f9924d05640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9908005180 0x7f990800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 -- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9908005180 msgr2=0x7f990800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9908005180 0x7f990800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 -- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9908003970 msgr2=0x7f9908003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9908003970 0x7f9908003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f9924d05640 1 -- 172.21.15.124:0/1664910497 reap_dead start 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.405+0000 7f991f7fe640 1 -- 172.21.15.124:0/1664910497 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9908003520 con 0x7f99080047c0 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f991f7fe640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 0x7f9908004c40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f992015fcf0 tx=0x7f99080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5658dd1a6ed5d854 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914005ad0 con 0x7f99080047c0 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9914005c30 con 0x7f99080047c0 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f991effd640 1 -- 172.21.15.124:0/1664910497 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991402ca50 con 0x7f99080047c0 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f9926768640 1 -- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 msgr2=0x7f9908004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.406+0000 7f9926768640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 0x7f9908004c40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f992015fcf0 tx=0x7f99080057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.407+0000 7f9926768640 1 -- 172.21.15.124:0/1664910497 shutdown_connections 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.407+0000 7f9926768640 1 --2- 172.21.15.124:0/1664910497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99080047c0 0x7f9908004c40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.407+0000 7f9926768640 1 -- 172.21.15.124:0/1664910497 >> 172.21.15.124:0/1664910497 conn(0x7f992006d9c0 msgr2=0x7f992006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.407+0000 7f9926768640 1 -- 172.21.15.124:0/1664910497 shutdown_connections 2023-12-13T23:56:22.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.407+0000 7f9926768640 1 -- 172.21.15.124:0/1664910497 wait complete. 2023-12-13T23:56:22.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.408+0000 7f9926768640 1 Processor -- start 2023-12-13T23:56:22.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.408+0000 7f9926768640 1 -- start start 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 0x7f992015dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f99200ad9a0 0x7f99200b1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99201f26d0 con 0x7f99200ade20 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99200ae3b0 con 0x7f99080047c0 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99201f2010 con 0x7f99200ad9a0 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49412/0 (socket says 172.21.15.124:49412) 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 -- 172.21.15.124:0/1527140063 learned_addr learned my addr 172.21.15.124:0/1527140063 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991f7fe640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f99200ad9a0 0x7f99200b1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991ffff640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 0x7f992015dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 -- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f99200ad9a0 msgr2=0x7f99200b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f99200ad9a0 0x7f99200b1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 -- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 msgr2=0x7f992015dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 0x7f992015dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 -- 172.21.15.124:0/1527140063 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99201f22f0 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991ffff640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 0x7f992015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9924d05640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f99200cc9e0 tx=0x7f991800fd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918003b80 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9918003d00 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918005310 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992015e1d0 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.409+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99201f0d00 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.410+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99201f1e70 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.411+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9918005470 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.411+0000 7f991cff9640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 0x7f990c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.411+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f99180897a0 con 0x7f99200ade20 2023-12-13T23:56:22.413 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.411+0000 7f991ffff640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 0x7f990c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.412+0000 7f991ffff640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 0x7f990c06d6b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9920050020 tx=0x7f9910008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.414 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.415+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f99201f1e70 con 0x7f99200ade20 2023-12-13T23:56:22.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.648+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f99201f2810 con 0x7f99200ade20 2023-12-13T23:56:22.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.650+0000 7f991cff9640 1 -- 172.21.15.124:0/1527140063 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f99201f2810 con 0x7f99200ade20 2023-12-13T23:56:22.648 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:56:22.648 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:56:22.648 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:56:22.649 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:56:22.649 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-12-13T23:56:22.649 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:56:22.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.651+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 msgr2=0x7f990c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.651+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 0x7f990c06d6b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9920050020 tx=0x7f9910008040 comp rx=0 tx=0).stop 2023-12-13T23:56:22.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.651+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 msgr2=0x7f99200b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.651+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f99200cc9e0 tx=0x7f991800fd40 comp rx=0 tx=0).stop 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 shutdown_connections 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f990c06b1f0 0x7f990c06d6b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99200ade20 0x7f99200b2150 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f99200ad9a0 0x7f99200b1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 --2- 172.21.15.124:0/1527140063 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f99080047c0 0x7f992015dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 >> 172.21.15.124:0/1527140063 conn(0x7f992006d9c0 msgr2=0x7f99200af890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 shutdown_connections 2023-12-13T23:56:22.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.652+0000 7f9926768640 1 -- 172.21.15.124:0/1527140063 wait complete. 2023-12-13T23:56:22.661 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:56:22.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.762+0000 7f7c5b6bb640 1 Processor -- start 2023-12-13T23:56:22.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.762+0000 7f7c5b6bb640 1 -- start start 2023-12-13T23:56:22.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c5b6bb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7c54143490 0x7f7c540b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c5b6bb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5415e2a0 con 0x7f7c5415abf0 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c5b6bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c5415b240 con 0x7f7c5415e730 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c5b6bb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c5415b520 con 0x7f7c54143490 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c59c58640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f7c5415abf0 0x7f7c540b2f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36228/0 (socket says 172.21.15.124:36228) 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c59c58640 1 -- 172.21.15.124:0/1267389917 learned_addr learned my addr 172.21.15.124:0/1267389917 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284879814 0 0) 0x7f7c5415e2a0 con 0x7f7c5415abf0 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c59457640 1 --2- 172.21.15.124:0/1267389917 >> v2:172.21.15.203:3300/0 conn(0x7f7c54143490 0x7f7c540b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.763+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c40003520 con 0x7f7c5415abf0 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3718466524 0 0) 0x7f7c40003520 con 0x7f7c5415abf0 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 >> v2:172.21.15.203:3300/0 conn(0x7f7c54143490 msgr2=0x7f7c540b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 --2- 172.21.15.124:0/1267389917 >> v2:172.21.15.203:3300/0 conn(0x7f7c54143490 0x7f7c540b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 >> v1:172.21.15.203:6789/0 conn(0x7f7c5415e730 legacy=0x7f7c5415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5415dc50 con 0x7f7c5415abf0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f7c4c001670 con 0x7f7c5415abf0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 >> v1:172.21.15.124:6789/0 conn(0x7f7c5415abf0 legacy=0x7f7c540b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c40003970 0x7f7c40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c40005180 0x7f7c4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5415b520 con 0x7f7c400047c0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4000c020 con 0x7f7c40003970 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4000e3f0 con 0x7f7c40005180 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c59457640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c40003970 0x7f7c40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c59c58640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c40005180 0x7f7c4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 -- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c40005180 msgr2=0x7f7c4000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c40005180 0x7f7c4000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 -- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c40003970 msgr2=0x7f7c40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c40003970 0x7f7c40003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c58c56640 1 -- 172.21.15.124:0/1267389917 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40003520 con 0x7f7c400047c0 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.764+0000 7f7c59c58640 1 -- 172.21.15.124:0/1267389917 reap_dead start 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c58c56640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c40004c40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7c44001750 tx=0x7f7c400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6eea6e6c067e0bb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c440052d0 con 0x7f7c400047c0 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4402e050 con 0x7f7c400047c0 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c4bfff640 1 -- 172.21.15.124:0/1267389917 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44003830 con 0x7f7c400047c0 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 msgr2=0x7f7c40004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.765+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c40004c40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f7c44001750 tx=0x7f7c400057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.766+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/1267389917 shutdown_connections 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.766+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/1267389917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c40004c40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.766+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/1267389917 >> 172.21.15.124:0/1267389917 conn(0x7f7c5406a4d0 msgr2=0x7f7c5406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.766+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/1267389917 shutdown_connections 2023-12-13T23:56:22.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.766+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/1267389917 wait complete. 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.767+0000 7f7c5b6bb640 1 Processor -- start 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.767+0000 7f7c5b6bb640 1 -- start start 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.767+0000 7f7c5b6bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 0x7f7c54141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 0x7f7c54141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c541eef90 con 0x7f7c400047c0 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5413a0b0 con 0x7f7c54159b60 2023-12-13T23:56:22.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5413a390 con 0x7f7c54139c10 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49430/0 (socket says 172.21.15.124:49430) 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 -- 172.21.15.124:0/54692544 learned_addr learned my addr 172.21.15.124:0/54692544 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59c58640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 0x7f7c54141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c58c56640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 0x7f7c54141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 -- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 msgr2=0x7f7c54141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 0x7f7c54141980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 -- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 msgr2=0x7f7c54141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 0x7f7c54141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 -- 172.21.15.124:0/54692544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c5413a670 con 0x7f7c400047c0 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59c58640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 0x7f7c54141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c58c56640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 0x7f7c54141980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c59457640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009f90 tx=0x7f7c5000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5000e040 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5000f040 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c541eebb0 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c50015870 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.768+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c5413e880 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.769+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c540b4380 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.770+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7c500159d0 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.770+0000 7f7c49ffb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 0x7f7c3806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.770+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c5008a7e0 con 0x7f7c400047c0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.771+0000 7f7c58c56640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 0x7f7c3806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:22.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.771+0000 7f7c58c56640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 0x7f7c3806d6b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7c44033000 tx=0x7f7c44009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:22.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.775+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7c500555a0 con 0x7f7c400047c0 2023-12-13T23:56:22.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.974+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c54069400 con 0x7f7c400047c0 2023-12-13T23:56:22.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.974+0000 7f7c49ffb640 1 -- 172.21.15.124:0/54692544 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f7c5000fea0 con 0x7f7c400047c0 2023-12-13T23:56:22.973 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:22.973 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:20.337597+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:56:22.974 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 14 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 msgr2=0x7f7c3806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 0x7f7c3806d6b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f7c44033000 tx=0x7f7c44009030 comp rx=0 tx=0).stop 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 msgr2=0x7f7c541396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f7c50009f90 tx=0x7f7c5000f580 comp rx=0 tx=0).stop 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 shutdown_connections 2023-12-13T23:56:22.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7c3806b1f0 0x7f7c3806d6b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7c54159b60 0x7f7c54141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7c54139c10 0x7f7c54141980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 --2- 172.21.15.124:0/54692544 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c400047c0 0x7f7c541396d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 >> 172.21.15.124:0/54692544 conn(0x7f7c5406a4d0 msgr2=0x7f7c54142a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 shutdown_connections 2023-12-13T23:56:22.976 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:22.976+0000 7f7c5b6bb640 1 -- 172.21.15.124:0/54692544 wait complete. 2023-12-13T23:56:23.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-13T23:56:18.320734+0000', 'modified': '2023-12-13T23:56:20.337597+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5271}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5271': {'gid': 5271, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.124:6835/4122494545', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.124:6834', 'nonce': 4122494545}, {'type': 'v1', 'addr': '172.21.15.124:6835', 'nonce': 4122494545}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701272860131327, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-13T23:56:23.388 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-13T23:56:23.389 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-12-13T23:56:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.491+0000 7fdafd856640 1 Processor -- start 2023-12-13T23:56:23.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.491+0000 7fdafd856640 1 -- start start 2023-12-13T23:56:23.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdafd856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdaf80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdafd856640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf80b2b10 con 0x7fdaf80d2d90 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdafd856640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf80d1590 con 0x7fdaf80d2910 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdafd856640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80d1870 con 0x7fdaf815d330 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdaf77fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdaf80d2d90 0x7fdaf80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36244/0 (socket says 172.21.15.124:36244) 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.492+0000 7fdaf77fe640 1 -- 172.21.15.124:0/1443982316 learned_addr learned my addr 172.21.15.124:0/1443982316 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdaf80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567146420 0 0) 0x7fdaf80b2b10 con 0x7fdaf80d2d90 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdae0003520 con 0x7fdaf80d2d90 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782900668 0 0) 0x7fdae0003520 con 0x7fdaf80d2d90 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 msgr2=0x7fdaf80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdaf80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 >> v1:172.21.15.203:6789/0 conn(0x7fdaf80d2910 legacy=0x7fdaf80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf80d1b50 con 0x7fdaf80d2d90 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fdaf0001670 con 0x7fdaf80d2d90 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 >> v1:172.21.15.124:6789/0 conn(0x7fdaf80d2d90 legacy=0x7fdaf80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdae00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0003f20 0x7fdae00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0004d00 0x7fdae0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80d1870 con 0x7fdae0004d00 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae000bb10 con 0x7fdae0003f20 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae000bcd0 con 0x7fdaf815d330 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0004d00 0x7fdae0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdae00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0003f20 0x7fdae00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 -- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 msgr2=0x7fdae00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf815d330 0x7fdae00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 -- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0003f20 msgr2=0x7fdae00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0003f20 0x7fdae00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.493+0000 7fdaf77fe640 1 -- 172.21.15.124:0/1443982316 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae0003520 con 0x7fdae0004d00 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.494+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0004d00 0x7fdae0005180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c210 tx=0x7fdae00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db89b00f6ae844a9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.494+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0005ad0 con 0x7fdae0004d00 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.494+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf0005c50 con 0x7fdae0004d00 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.494+0000 7fdaf5ffb640 1 -- 172.21.15.124:0/1443982316 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf002c890 con 0x7fdae0004d00 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 -- 172.21.15.124:0/1443982316 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0004d00 msgr2=0x7fdae0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 --2- 172.21.15.124:0/1443982316 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0004d00 0x7fdae0005180 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c210 tx=0x7fdae00057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdaf77fe640 1 -- 172.21.15.124:0/1443982316 reap_dead start 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 -- 172.21.15.124:0/1443982316 shutdown_connections 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 -- 172.21.15.124:0/1443982316 >> 172.21.15.124:0/1443982316 conn(0x7fdaf8072020 msgr2=0x7fdaf8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 -- 172.21.15.124:0/1443982316 shutdown_connections 2023-12-13T23:56:23.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.495+0000 7fdafd856640 1 -- 172.21.15.124:0/1443982316 wait complete. 2023-12-13T23:56:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.496+0000 7fdafd856640 1 Processor -- start 2023-12-13T23:56:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.496+0000 7fdafd856640 1 -- start start 2023-12-13T23:56:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0004d00 0x7fdaf81ecdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf80ca480 0x7fdaf80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf8159cf0 con 0x7fdaf8159330 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80cd6b0 con 0x7fdae0004d00 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf80cc610 con 0x7fdaf80ca480 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49440/0 (socket says 172.21.15.124:49440) 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 -- 172.21.15.124:0/85198373 learned_addr learned my addr 172.21.15.124:0/85198373 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf80ca480 0x7fdaf80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0004d00 0x7fdaf81ecdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 -- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf80ca480 msgr2=0x7fdaf80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf80ca480 0x7fdaf80ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 -- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0004d00 msgr2=0x7fdaf81ecdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0004d00 0x7fdaf81ecdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 -- 172.21.15.124:0/85198373 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf80cc8f0 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00f820 tx=0x7fdaec00fce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdadffff640 1 -- 172.21.15.124:0/85198373 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec0050d0 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdadffff640 1 -- 172.21.15.124:0/85198373 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec005230 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.497+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf80ccbd0 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.498+0000 7fdadffff640 1 -- 172.21.15.124:0/85198373 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec00d730 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.498+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf80cd190 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.499+0000 7fdadffff640 1 -- 172.21.15.124:0/85198373 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdaec00d890 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.499+0000 7fdadffff640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 0x7fdae406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.499+0000 7fdadffff640 1 -- 172.21.15.124:0/85198373 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdaec0898f0 con 0x7fdaf8159330 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.500+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 0x7fdae406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.500+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 0x7fdae406d680 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80c9db0 tx=0x7fdae800b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.500+0000 7fdb003ea740 1 Processor -- start 2023-12-13T23:56:23.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.500+0000 7fdb003ea740 1 -- start start 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1b7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1b8000 0x557f8d1e7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d12b020 con 0x557f8d1e4c70 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d128710 con 0x557f8d1e4890 2023-12-13T23:56:23.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f8d124000 con 0x557f8d1b8000 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdaf67fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdaf67fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49452/0 (socket says 172.21.15.124:49452) 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdaf67fc640 1 -- 172.21.15.124:0/1314145100 learned_addr learned my addr 172.21.15.124:0/1314145100 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1b7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.501+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1b8000 0x557f8d1e7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 -- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1b8000 msgr2=0x557f8d1e7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1b8000 0x557f8d1e7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 -- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 msgr2=0x557f8d1b7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1b7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 -- 172.21.15.124:0/1314145100 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f8d0e6c90 con 0x557f8d1e4c70 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdaec087ed0 tx=0x7fdaec057d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996fe0981a475e84 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdadd7fa640 1 -- 172.21.15.124:0/1314145100 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec060670 con 0x557f8d1e4c70 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdadd7fa640 1 -- 172.21.15.124:0/1314145100 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec012e30 con 0x557f8d1e4c70 2023-12-13T23:56:23.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdadd7fa640 1 -- 172.21.15.124:0/1314145100 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaec05c160 con 0x557f8d1e4c70 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdb003ea740 1 -- 172.21.15.124:0/1314145100 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 msgr2=0x557f8d1e7620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.502+0000 7fdb003ea740 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fdaec087ed0 tx=0x7fdaec057d90 comp rx=0 tx=0).stop 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 -- 172.21.15.124:0/1314145100 shutdown_connections 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1b8000 0x557f8d1e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4c70 0x557f8d1e7620 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 --2- 172.21.15.124:0/1314145100 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1b7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 -- 172.21.15.124:0/1314145100 >> 172.21.15.124:0/1314145100 conn(0x557f8d1b9d30 msgr2=0x557f8d202620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 -- 172.21.15.124:0/1314145100 shutdown_connections 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 -- 172.21.15.124:0/1314145100 wait complete. 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 -- start start 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.503+0000 7fdb003ea740 1 Processor -- start 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1ea070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d205f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d0dde00 con 0x557f8d1fa380 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d0ddaa0 con 0x557f8d1e4890 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f8d0a3270 con 0x557f8d1e4c70 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdb003ea740 10 client.0 _ll_register_callbacks cb 0x557f8ce35ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49466/0 (socket says 172.21.15.124:49466) 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 -- 172.21.15.124:0/2400993340 learned_addr learned my addr 172.21.15.124:0/2400993340 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1ea070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d205f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 -- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 msgr2=0x557f8d205f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d205f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 -- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 msgr2=0x557f8d1ea070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1ea070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 -- 172.21.15.124:0/2400993340 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf80cd190 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c330 tx=0x7fdaf002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=642d449b1f32d4ce server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.504+0000 7fdad67fc640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf002cca0 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdaf003c070 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557f8d11ce90 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdaf002ce00 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 10 client.5424 fetch_fsmap learned FSMap version 14 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x557f8d122680 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f8d11ce90 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 14) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdaf00047e0 con 0x557f8d1fa380 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 10 client.5424 fetch_fsmap finished waiting for FSMap version 14 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 10 client.5424 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 20 client.5424 populate_metadata read hostname 'smithi124' 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1fb180 0x557f8d1fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 4 client.5424 mds_command: new command op to 5271 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 --> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557f8d11cbb0 con 0x557f8d1fb180 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1fb180 0x557f8d1fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdad67fc640 10 client.5424 ms_handle_connect on v2:172.21.15.124:6834/4122494545 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.505+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1fb180 0x557f8d1fb560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a554651efa26cebd server_cookie=cd5a8d83329a53ff in_seq=0 out_seq=0 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdad67fc640 1 -- 172.21.15.124:0/2400993340 <== mds.0 v2:172.21.15.124:6834/4122494545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557f8d11cbb0 con 0x557f8d1fb180 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdad67fc640 10 client.5424 handle_command_reply: tid=0 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 client.5424 shutdown 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 msgr2=0x557f8d1e3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fdaf000c330 tx=0x7fdaf002e070 comp rx=0 tx=0).stop 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 shutdown_connections 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1fb180 0x557f8d1fb560 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d1e3090 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d205f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 --2- 172.21.15.124:0/2400993340 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d1ea070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdad67fc640 0 client.5424 ms_handle_reset on v2:172.21.15.124:6834/4122494545 2023-12-13T23:56:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 >> 172.21.15.124:0/2400993340 conn(0x557f8d1b8000 msgr2=0x557f8d1fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 shutdown_connections 2023-12-13T23:56:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 1 -- 172.21.15.124:0/2400993340 wait complete. 2023-12-13T23:56:23.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.506+0000 7fdb003ea740 20 client.5424 trim_cache size 0 max 16384 2023-12-13T23:56:23.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.522+0000 7fdb003ea740 1 Processor -- start 2023-12-13T23:56:23.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.522+0000 7fdb003ea740 1 -- start start 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4c70 0x557f8d1e2b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1fa380 0x557f8d1bc530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d205090 con 0x557f8d1e4890 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d1eb530 con 0x557f8d1e4c70 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f8d2035b0 con 0x557f8d1fa380 2023-12-13T23:56:23.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49476/0 (socket says 172.21.15.124:49476) 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 -- 172.21.15.124:0/3752981760 learned_addr learned my addr 172.21.15.124:0/3752981760 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 -- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1fa380 msgr2=0x557f8d1bc530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1fa380 0x557f8d1bc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 -- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4c70 msgr2=0x557f8d1e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4c70 0x557f8d1e2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 -- 172.21.15.124:0/3752981760 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f8d0bfd00 con 0x557f8d1e4890 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdae8015a90 tx=0x7fdae8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f142201c9393c1c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaddffb640 1 -- 172.21.15.124:0/3752981760 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8029040 con 0x557f8d1e4890 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaddffb640 1 -- 172.21.15.124:0/3752981760 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae800f040 con 0x557f8d1e4890 2023-12-13T23:56:23.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.523+0000 7fdaddffb640 1 -- 172.21.15.124:0/3752981760 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae800e040 con 0x557f8d1e4890 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- 172.21.15.124:0/3752981760 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 msgr2=0x557f8d1e9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fdae8015a90 tx=0x7fdae8019000 comp rx=0 tx=0).stop 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- 172.21.15.124:0/3752981760 shutdown_connections 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4c70 0x557f8d1e2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1fa380 0x557f8d1bc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3752981760 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1e4890 0x557f8d1e9950 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- 172.21.15.124:0/3752981760 >> 172.21.15.124:0/3752981760 conn(0x557f8d1b8000 msgr2=0x557f8d1bd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- 172.21.15.124:0/3752981760 shutdown_connections 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- 172.21.15.124:0/3752981760 wait complete. 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.524+0000 7fdb003ea740 1 -- start start 2023-12-13T23:56:23.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 Processor -- start 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d26c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d2042c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d1e8810 con 0x557f8d1fa380 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f8d26cf20 con 0x557f8d1e4890 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f8d222f40 con 0x557f8d1e4c70 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdb003ea740 10 client.0 _ll_register_callbacks cb 0x7fdaec0078d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdaf77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdaf77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49484/0 (socket says 172.21.15.124:49484) 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdaf77fe640 1 -- 172.21.15.124:0/3581241680 learned_addr learned my addr 172.21.15.124:0/3581241680 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d26c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.525+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d2042c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 -- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 msgr2=0x557f8d2042c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d2042c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 -- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 msgr2=0x557f8d26c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d26c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 -- 172.21.15.124:0/3581241680 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae8015850 con 0x557f8d1fa380 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf67fc640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d2042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdaf77fe640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0007f30 tx=0x7fdaf0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb6bfe5ff0667bc8 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:56:23.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf0000f00 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdaf003c070 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdaec012e30 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdaf002c570 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 10 client.5430 fetch_fsmap learned FSMap version 14 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0x557f8d0e54f0 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaec012e30 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 14) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fdaf002ccb0 con 0x557f8d1fa380 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 10 client.5430 fetch_fsmap finished waiting for FSMap version 14 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 10 client.5430 resolve_mds: resolved name 'a' to daemon mds.a 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 20 client.5430 populate_metadata read hostname 'smithi124' 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1e9830 0x557f8d1e9c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 4 client.5430 mds_command: new command op to 5271 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.526+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 --> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557f8ce246c0 con 0x557f8d1e9830 2023-12-13T23:56:23.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1e9830 0x557f8d1e9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdaf6ffd640 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1e9830 0x557f8d1e9c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c58dc46ea682ae92 server_cookie=303bef321cba6f3a in_seq=0 out_seq=0 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdad67fc640 10 client.5430 ms_handle_connect on v2:172.21.15.124:6834/4122494545 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdad67fc640 1 -- 172.21.15.124:0/3581241680 <== mds.0 v2:172.21.15.124:6834/4122494545 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557f8ce246c0 con 0x557f8d1e9830 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdad67fc640 10 client.5430 handle_command_reply: tid=0 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 client.5430 shutdown 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 msgr2=0x557f8d222330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fdaf0007f30 tx=0x7fdaf0001a00 comp rx=0 tx=0).stop 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 shutdown_connections 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:6834/4122494545,v1:172.21.15.124:6835/4122494545] conn(0x557f8d1e9830 0x557f8d1e9c10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557f8d1fa380 0x557f8d222330 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x557f8d1e4c70 0x557f8d2042c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdad67fc640 0 client.5430 ms_handle_reset on v2:172.21.15.124:6834/4122494545 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 --2- 172.21.15.124:0/3581241680 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x557f8d1e4890 0x557f8d26c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 >> 172.21.15.124:0/3581241680 conn(0x557f8d1b8000 msgr2=0x557f8d1e2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 shutdown_connections 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 1 -- 172.21.15.124:0/3581241680 wait complete. 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.527+0000 7fdb003ea740 20 client.5430 trim_cache size 0 max 16384 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "cluster_fsid": "f45f9ea9-91cf-4589-b298-b4d7e1fc4f09", 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "whoami": 0, 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5271, 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "want_state": "up:active", 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "state": "up:active", 2023-12-13T23:56:23.533 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs", 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout: "rank_uptime": 4.5420277130000004, 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout: "mdsmap_epoch": 14, 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch": 35, 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch_barrier": 35, 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout: "uptime": 12.484742014 2023-12-13T23:56:23.534 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 msgr2=0x7fdae406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 0x7fdae406d680 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fdaf80c9db0 tx=0x7fdae800b040 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 msgr2=0x7fdaf81597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fdaec00f820 tx=0x7fdaec00fce0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 shutdown_connections 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdae406b1c0 0x7fdae406d680 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdaf80ca480 0x7fdaf80ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdaf8159330 0x7fdaf81597b0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 --2- 172.21.15.124:0/85198373 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0004d00 0x7fdaf81ecdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 >> 172.21.15.124:0/85198373 conn(0x7fdaf8072020 msgr2=0x7fdaf80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 shutdown_connections 2023-12-13T23:56:23.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.528+0000 7fdafd856640 1 -- 172.21.15.124:0/85198373 wait complete. 2023-12-13T23:56:23.537 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:56:23.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 Processor -- start 2023-12-13T23:56:23.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 -- start start 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbd680b1400 0x7fbd6815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 0x7fbd6815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6815e2a0 con 0x7fbd68143490 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6815b520 con 0x7fbd680b1400 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.638+0000 7fbd6eabb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd68143870 con 0x7fbd6815e730 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbd680b1400 0x7fbd6815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbd680b1400 0x7fbd6815a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49498/0 (socket says 172.21.15.124:49498) 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 -- 172.21.15.124:0/4236121185 learned_addr learned my addr 172.21.15.124:0/4236121185 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd6d058640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 0x7fbd6815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 -- 172.21.15.124:0/4236121185 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 msgr2=0x7fbd6815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 0x7fbd6815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 -- 172.21.15.124:0/4236121185 >> v1:172.21.15.203:6789/0 conn(0x7fbd68143490 legacy=0x7fbd680b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:23.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd67fff640 1 -- 172.21.15.124:0/4236121185 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6815b240 con 0x7fbd680b1400 2023-12-13T23:56:23.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.639+0000 7fbd6d058640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 0x7fbd6815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd67fff640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd680b1400 0x7fbd6815a4e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009870 tx=0x7fbd5402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d497d786a7318aef server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd677fe640 1 -- 172.21.15.124:0/4236121185 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54030840 con 0x7fbd680b1400 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd677fe640 1 -- 172.21.15.124:0/4236121185 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd540309a0 con 0x7fbd680b1400 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd677fe640 1 -- 172.21.15.124:0/4236121185 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54030d30 con 0x7fbd680b1400 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 -- 172.21.15.124:0/4236121185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd680b1400 msgr2=0x7fbd6815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd680b1400 0x7fbd6815a4e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009870 tx=0x7fbd5402b980 comp rx=0 tx=0).stop 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 -- 172.21.15.124:0/4236121185 shutdown_connections 2023-12-13T23:56:23.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd6815e730 0x7fbd6815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/4236121185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd680b1400 0x7fbd6815a4e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 -- 172.21.15.124:0/4236121185 >> 172.21.15.124:0/4236121185 conn(0x7fbd6806a4d0 msgr2=0x7fbd6806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.640+0000 7fbd6eabb640 1 -- 172.21.15.124:0/4236121185 shutdown_connections 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.641+0000 7fbd6eabb640 1 -- 172.21.15.124:0/4236121185 wait complete. 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.642+0000 7fbd6eabb640 1 Processor -- start 2023-12-13T23:56:23.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.642+0000 7fbd6eabb640 1 -- start start 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.642+0000 7fbd6eabb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd680b1400 0x7fbd68131970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.642+0000 7fbd6eabb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.642+0000 7fbd6eabb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd6815e730 0x7fbd680add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6eabb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6812f1c0 con 0x7fbd68143490 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6eabb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6812e5b0 con 0x7fbd6815e730 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6eabb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6812e890 con 0x7fbd680b1400 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49512/0 (socket says 172.21.15.124:49512) 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 -- 172.21.15.124:0/3301696867 learned_addr learned my addr 172.21.15.124:0/3301696867 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6d058640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd6815e730 0x7fbd680add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6c857640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd680b1400 0x7fbd68131970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 -- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd680b1400 msgr2=0x7fbd68131970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd680b1400 0x7fbd68131970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 -- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd6815e730 msgr2=0x7fbd680add10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd6815e730 0x7fbd680add10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 -- 172.21.15.124:0/3301696867 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd540094d0 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd67fff640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbd54000c00 tx=0x7fbd54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd54030810 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54043400 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6812eb70 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd540437d0 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.643+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6815d4d0 con 0x7fbd68143490 2023-12-13T23:56:23.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.644+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6812f330 con 0x7fbd68143490 2023-12-13T23:56:23.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.645+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbd54043930 con 0x7fbd68143490 2023-12-13T23:56:23.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.645+0000 7fbd657fa640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 0x7fbd4c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:23.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.645+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd5403e070 con 0x7fbd68143490 2023-12-13T23:56:23.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.645+0000 7fbd6c857640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 0x7fbd4c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:23.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.646+0000 7fbd6c857640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 0x7fbd4c06d680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c000f60 tx=0x7fbd5c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:23.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.649+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbd540779f0 con 0x7fbd68143490 2023-12-13T23:56:23.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.850+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd68069400 con 0x7fbd68143490 2023-12-13T23:56:23.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.850+0000 7fbd657fa640 1 -- 172.21.15.124:0/3301696867 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v14) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbd5407b010 con 0x7fbd68143490 2023-12-13T23:56:23.849 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:56:23.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 msgr2=0x7fbd4c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 0x7fbd4c06d680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c000f60 tx=0x7fbd5c009040 comp rx=0 tx=0).stop 2023-12-13T23:56:23.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 msgr2=0x7fbd6815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:23.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fbd54000c00 tx=0x7fbd54004730 comp rx=0 tx=0).stop 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 shutdown_connections 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbd4c06b1c0 0x7fbd4c06d680 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbd6815e730 0x7fbd680add10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbd68143490 0x7fbd6815cf90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 --2- 172.21.15.124:0/3301696867 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbd680b1400 0x7fbd68131970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 >> 172.21.15.124:0/3301696867 conn(0x7fbd6806a4d0 msgr2=0x7fbd6806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 shutdown_connections 2023-12-13T23:56:23.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:23.852+0000 7fbd6eabb640 1 -- 172.21.15.124:0/3301696867 wait complete. 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:56:23.862 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:23.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:23.863 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:56:23.878 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:56:23.878 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:56:23.879 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42623sec preferred_lft 42623sec 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:23.880 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:23.881 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:56:23.881 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:23.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:23.881 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:56:23.944 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:56:23.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:23.946 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:56:24.002 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.002 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:56:24.003 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:56:24.003 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:56:24.003 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:56:24.003 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:56:24.003 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:56:24.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.097 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:56:24.111 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:56:24.111 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:56:24.111 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:24.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.112 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:56:24.178 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:56:24.179 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:24.179 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:24.233 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:24.233 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:24.233 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:24.287 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:24.354 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:24.410 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:24.411 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:24.412 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:24.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.413 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:24.465 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:56:24.465 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:56:24.508 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:24.534 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:24.589 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:24.590 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:24.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.592 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:24.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:56:24.593+0000 7efd6677c480 -1 init, newargv = 0x55c0f11038b0 newargc=15 2023-12-13T23:56:24.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[103467]: starting ceph client 2023-12-13T23:56:24.596 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[103467]: starting fuse 2023-12-13T23:56:24.607 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:56:24.607 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:24.607 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:56:24.608 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:56:24.609 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:56:24.609 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:24.609 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:56:24.609 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:56:24.727 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.103467.asok 2023-12-13T23:56:24.730 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:24 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:56:24.737 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.103467.asok 2023-12-13T23:56:24.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.737 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.103467.asok status 2023-12-13T23:56:24.872 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "103467", 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5445, 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:56:24.873 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5445 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 545330818 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 545330818 2023-12-13T23:56:24.874 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5445 192.168.0.1:0/545330818", 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/545330818", 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 14, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 35, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:56:24.875 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:56:24.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.881 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:56:24.881 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:24.896 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:56:24.896 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:24.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:24.896 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:24.964 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2023-12-13T23:56:25.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 Processor -- start 2023-12-13T23:56:25.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 -- start start 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe50415d330 0x7fe5040b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040b2b10 con 0x7fe5040d2910 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5040d1590 con 0x7fe5040d3900 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.109+0000 7fe50924b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5040d1870 con 0x7fe50415d330 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe502d9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe50415d330 0x7fe5040b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe4fbfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe5040d2910 0x7fe5040d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36250/0 (socket says 172.21.15.124:36250) 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe4fbfff640 1 -- 172.21.15.124:0/1107707061 learned_addr learned my addr 172.21.15.124:0/1107707061 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe502d9d640 1 -- 172.21.15.124:0/1107707061 >> v1:172.21.15.203:6789/0 conn(0x7fe5040d3900 legacy=0x7fe5040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe502d9d640 1 -- 172.21.15.124:0/1107707061 >> v1:172.21.15.124:6789/0 conn(0x7fe5040d2910 legacy=0x7fe5040d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe502d9d640 1 -- 172.21.15.124:0/1107707061 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5040d1b50 con 0x7fe50415d330 2023-12-13T23:56:25.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.110+0000 7fe50259c640 1 -- 172.21.15.124:0/1107707061 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748571080 0 0) 0x7fe5040b2b10 con 0x7fe5040d2910 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe502d9d640 1 --2- 172.21.15.124:0/1107707061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5040b8b80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40097a0 tx=0x7fe4f402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d245c7b6fd61c813 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50259c640 1 -- 172.21.15.124:0/1107707061 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f402e070 con 0x7fe50415d330 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50259c640 1 -- 172.21.15.124:0/1107707061 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f40315a0 con 0x7fe50415d330 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50259c640 1 -- 172.21.15.124:0/1107707061 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f4031930 con 0x7fe50415d330 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 -- 172.21.15.124:0/1107707061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 msgr2=0x7fe5040b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 --2- 172.21.15.124:0/1107707061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5040b8b80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fe4f40097a0 tx=0x7fe4f402b840 comp rx=0 tx=0).stop 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 -- 172.21.15.124:0/1107707061 shutdown_connections 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 --2- 172.21.15.124:0/1107707061 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5040b8b80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 -- 172.21.15.124:0/1107707061 >> 172.21.15.124:0/1107707061 conn(0x7fe504072020 msgr2=0x7fe504072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 -- 172.21.15.124:0/1107707061 shutdown_connections 2023-12-13T23:56:25.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.111+0000 7fe50924b640 1 -- 172.21.15.124:0/1107707061 wait complete. 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 Processor -- start 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 -- start start 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 0x7fe5040ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 0x7fe504148fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040cd750 con 0x7fe50415d330 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5041f28f0 con 0x7fe5040d3900 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50924b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5040ca870 con 0x7fe5040d2910 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50359e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50359e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49538/0 (socket says 172.21.15.124:49538) 2023-12-13T23:56:25.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.113+0000 7fe50359e640 1 -- 172.21.15.124:0/2042775277 learned_addr learned my addr 172.21.15.124:0/2042775277 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe4fbfff640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 0x7fe504148fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 -- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 msgr2=0x7fe5040ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe502d9d640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 0x7fe5040ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 0x7fe5040ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 -- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 msgr2=0x7fe504148fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 0x7fe504148fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 -- 172.21.15.124:0/2042775277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f4009400 con 0x7fe50415d330 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe4fbfff640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 0x7fe504148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe502d9d640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 0x7fe5040ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50359e640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c730 tx=0x7fe4fc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00a9f0 con 0x7fe50415d330 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4fc00ab50 con 0x7fe50415d330 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00ce40 con 0x7fe50415d330 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5040ccb40 con 0x7fe50415d330 2023-12-13T23:56:25.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.114+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5040cd100 con 0x7fe50415d330 2023-12-13T23:56:25.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.115+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5040d2dc0 con 0x7fe50415d330 2023-12-13T23:56:25.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.115+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe4fc012420 con 0x7fe50415d330 2023-12-13T23:56:25.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.116+0000 7fe4fb7fe640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 0x7fe4e406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:25.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.116+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4fc089e50 con 0x7fe50415d330 2023-12-13T23:56:25.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.116+0000 7fe502d9d640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 0x7fe4e406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:25.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.116+0000 7fe502d9d640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 0x7fe4e406d680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4036000 tx=0x7fe4f403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:25.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.120+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe4fc054c10 con 0x7fe50415d330 2023-12-13T23:56:25.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:25.287+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) v1 -- 0x7fe504069400 con 0x7fe50415d330 2023-12-13T23:56:26.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.025+0000 7fe4fb7fe640 1 -- 172.21.15.124:0/2042775277 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v36) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe4fc016020 con 0x7fe50415d330 2023-12-13T23:56:26.024 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'data_new' created 2023-12-13T23:56:26.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.028+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 msgr2=0x7fe4e406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.028+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 0x7fe4e406d680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe4f4036000 tx=0x7fe4f403d040 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.028+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 msgr2=0x7fe5041f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.028+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc00c730 tx=0x7fe4fc00cbf0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.028+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 shutdown_connections 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4e406b1c0 0x7fe4e406d680 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5040d3900 0x7fe504148fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe50415d330 0x7fe5041f5fb0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 --2- 172.21.15.124:0/2042775277 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe5040d2910 0x7fe5040ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 >> 172.21.15.124:0/2042775277 conn(0x7fe504072020 msgr2=0x7fe5040b7150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 shutdown_connections 2023-12-13T23:56:26.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.029+0000 7fe50924b640 1 -- 172.21.15.124:0/2042775277 wait complete. 2023-12-13T23:56:26.042 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2023-12-13T23:56:26.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.147+0000 7ff96233c640 1 Processor -- start 2023-12-13T23:56:26.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.147+0000 7ff96233c640 1 -- start start 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff95c143490 0x7ff95c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff95c15a4e0 0x7ff95c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c15b520 con 0x7ff95c143490 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c143870 con 0x7ff95c15a4e0 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff96233c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c15e2a0 con 0x7ff95c15e730 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff95c143490 0x7ff95c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff95c143490 0x7ff95c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49548/0 (socket says 172.21.15.124:49548) 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 -- 172.21.15.124:0/2013880043 learned_addr learned my addr 172.21.15.124:0/2013880043 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95b7fe640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff9608d9640 1 --2- 172.21.15.124:0/2013880043 >> v2:172.21.15.203:3300/0 conn(0x7ff95c15a4e0 0x7ff95c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 -- 172.21.15.124:0/2013880043 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 msgr2=0x7ff95c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 -- 172.21.15.124:0/2013880043 >> v2:172.21.15.203:3300/0 conn(0x7ff95c15a4e0 msgr2=0x7ff95c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 --2- 172.21.15.124:0/2013880043 >> v2:172.21.15.203:3300/0 conn(0x7ff95c15a4e0 0x7ff95c15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.148+0000 7ff95bfff640 1 -- 172.21.15.124:0/2013880043 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c15b240 con 0x7ff95c143490 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff95bfff640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c143490 0x7ff95c0b0cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff95c15eb90 tx=0x7ff94c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8ddd21892a4a31c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff95affd640 1 -- 172.21.15.124:0/2013880043 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031870 con 0x7ff95c143490 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff95affd640 1 -- 172.21.15.124:0/2013880043 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c0319d0 con 0x7ff95c143490 2023-12-13T23:56:26.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff95affd640 1 -- 172.21.15.124:0/2013880043 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c031d60 con 0x7ff95c143490 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff96233c640 1 -- 172.21.15.124:0/2013880043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c143490 msgr2=0x7ff95c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.149+0000 7ff96233c640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c143490 0x7ff95c0b0cf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff95c15eb90 tx=0x7ff94c02b910 comp rx=0 tx=0).stop 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 -- 172.21.15.124:0/2013880043 shutdown_connections 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 --2- 172.21.15.124:0/2013880043 >> v2:172.21.15.203:3300/0 conn(0x7ff95c15a4e0 0x7ff95c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 --2- 172.21.15.124:0/2013880043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c143490 0x7ff95c0b0cf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 -- 172.21.15.124:0/2013880043 >> 172.21.15.124:0/2013880043 conn(0x7ff95c06a4d0 msgr2=0x7ff95c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 -- 172.21.15.124:0/2013880043 shutdown_connections 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.150+0000 7ff96233c640 1 -- 172.21.15.124:0/2013880043 wait complete. 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.151+0000 7ff96233c640 1 Processor -- start 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.151+0000 7ff96233c640 1 -- start start 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95c143490 0x7ff95c13ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c1eec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c140a30 con 0x7ff95c15a4e0 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c13f5d0 con 0x7ff95c143490 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c140090 con 0x7ff95c15e730 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49556/0 (socket says 172.21.15.124:49556) 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 -- 172.21.15.124:0/948122260 learned_addr learned my addr 172.21.15.124:0/948122260 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95bfff640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95c143490 0x7ff95c13ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff9608d9640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c1eec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 -- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 msgr2=0x7ff95c1eec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c1eec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 -- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95c143490 msgr2=0x7ff95c13ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95c143490 0x7ff95c13ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 -- 172.21.15.124:0/948122260 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c0094d0 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff9608d9640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff95b7fe640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff95000d5f0 tx=0x7ff95000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9500125f0 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950012750 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c140370 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950012b00 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.152+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c1592b0 con 0x7ff95c15a4e0 2023-12-13T23:56:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.153+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c140ca0 con 0x7ff95c15a4e0 2023-12-13T23:56:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.154+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff950012c60 con 0x7ff95c15a4e0 2023-12-13T23:56:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.154+0000 7ff958ff9640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 0x7ff94806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.154+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6072+0+0 (secure 0 0 0) 0x7ff950089d80 con 0x7ff95c15a4e0 2023-12-13T23:56:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.155+0000 7ff95bfff640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 0x7ff94806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.155+0000 7ff95bfff640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 0x7ff94806d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff94c036000 tx=0x7ff94c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:26.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.159+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff9500549d0 con 0x7ff95c15a4e0 2023-12-13T23:56:26.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:26.365+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) v1 -- 0x7ff95c069400 con 0x7ff95c15a4e0 2023-12-13T23:56:27.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.037+0000 7ff958ff9640 1 -- 172.21.15.124:0/948122260 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 6 to fsmap v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7ff950057ff0 con 0x7ff95c15a4e0 2023-12-13T23:56:27.036 INFO:teuthology.orchestra.run.smithi124.stderr:added data pool 6 to fsmap 2023-12-13T23:56:27.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 msgr2=0x7ff94806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 0x7ff94806d6b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff94c036000 tx=0x7ff94c03c040 comp rx=0 tx=0).stop 2023-12-13T23:56:27.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 msgr2=0x7ff95c158d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ff95000d5f0 tx=0x7ff95000dab0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 shutdown_connections 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff94806b1f0 0x7ff94806d6b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95c15e730 0x7ff95c1eec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff95c15a4e0 0x7ff95c158d70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 --2- 172.21.15.124:0/948122260 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95c143490 0x7ff95c13ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 >> 172.21.15.124:0/948122260 conn(0x7ff95c06a4d0 msgr2=0x7ff95c070c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 shutdown_connections 2023-12-13T23:56:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.042+0000 7ff96233c640 1 -- 172.21.15.124:0/948122260 wait complete. 2023-12-13T23:56:27.053 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:56:27.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.157+0000 7fb194d61640 1 Processor -- start 2023-12-13T23:56:27.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.157+0000 7fb194d61640 1 -- start start 2023-12-13T23:56:27.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1901595c0 0x7fb19006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1901599a0 0x7fb19006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb1900b3a50 0x7fb190066270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb19006f8f0 con 0x7fb1901599a0 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900b2c90 con 0x7fb1900b3a50 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb194d61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900b2f70 con 0x7fb1901595c0 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb187fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1901599a0 0x7fb19006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb187fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1901599a0 0x7fb19006ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49564/0 (socket says 172.21.15.124:49564) 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb187fff640 1 -- 172.21.15.124:0/2270081326 learned_addr learned my addr 172.21.15.124:0/2270081326 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb18e59c640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1901595c0 0x7fb19006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.158+0000 7fb18ed9d640 1 --2- 172.21.15.124:0/2270081326 >> v2:172.21.15.203:3300/0 conn(0x7fb1900b3a50 0x7fb190066270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 -- 172.21.15.124:0/2270081326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1901595c0 msgr2=0x7fb19006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1901595c0 0x7fb19006b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 -- 172.21.15.124:0/2270081326 >> v2:172.21.15.203:3300/0 conn(0x7fb1900b3a50 msgr2=0x7fb190066270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 --2- 172.21.15.124:0/2270081326 >> v2:172.21.15.203:3300/0 conn(0x7fb1900b3a50 0x7fb190066270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 -- 172.21.15.124:0/2270081326 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900b3250 con 0x7fb1901599a0 2023-12-13T23:56:27.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.159+0000 7fb187fff640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006ba20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb1900b4380 tx=0x7fb17802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8297e73f7b7ba38 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb18dd9b640 1 -- 172.21.15.124:0/2270081326 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17802ee90 con 0x7fb1901599a0 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb18dd9b640 1 -- 172.21.15.124:0/2270081326 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb178036440 con 0x7fb1901599a0 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb18dd9b640 1 -- 172.21.15.124:0/2270081326 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1780367d0 con 0x7fb1901599a0 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 -- 172.21.15.124:0/2270081326 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 msgr2=0x7fb19006ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006ba20 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb1900b4380 tx=0x7fb17802e0e0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 -- 172.21.15.124:0/2270081326 shutdown_connections 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006ba20 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 --2- 172.21.15.124:0/2270081326 >> v2:172.21.15.203:3300/0 conn(0x7fb1900b3a50 0x7fb190066270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 --2- 172.21.15.124:0/2270081326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1901595c0 0x7fb19006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 -- 172.21.15.124:0/2270081326 >> 172.21.15.124:0/2270081326 conn(0x7fb190070e70 msgr2=0x7fb190071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 -- 172.21.15.124:0/2270081326 shutdown_connections 2023-12-13T23:56:27.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.160+0000 7fb194d61640 1 -- 172.21.15.124:0/2270081326 wait complete. 2023-12-13T23:56:27.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 Processor -- start 2023-12-13T23:56:27.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 -- start start 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1900b3a50 0x7fb19014d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1901595c0 0x7fb1901eeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014d8f0 con 0x7fb1901599a0 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb190151800 con 0x7fb1901595c0 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb194d61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb190150bf0 con 0x7fb1900b3a50 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb18ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb18ed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49570/0 (socket says 172.21.15.124:49570) 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.162+0000 7fb18ed9d640 1 -- 172.21.15.124:0/3784865532 learned_addr learned my addr 172.21.15.124:0/3784865532 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18e59c640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1900b3a50 0x7fb19014d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb187fff640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1901595c0 0x7fb1901eeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 -- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1900b3a50 msgr2=0x7fb19014d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1900b3a50 0x7fb19014d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 -- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1901595c0 msgr2=0x7fb1901eeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.163 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1901595c0 0x7fb1901eeca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 -- 172.21.15.124:0/3784865532 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17800b770 con 0x7fb1901599a0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb18ed9d640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb188007e60 tx=0x7fb18800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb188014620 con 0x7fb1901599a0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb18800ee90 con 0x7fb1901599a0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1880149e0 con 0x7fb1901599a0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb190150ed0 con 0x7fb1901599a0 2023-12-13T23:56:27.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.163+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb190151490 con 0x7fb1901599a0 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.164+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb190151970 con 0x7fb1901599a0 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.165+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb188014b40 con 0x7fb1901599a0 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.165+0000 7fb186ffd640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 0x7fb17006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.165+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb18808a2d0 con 0x7fb1901599a0 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.165+0000 7fb18e59c640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 0x7fb17006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.166+0000 7fb18e59c640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 0x7fb17006d680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb180006fd0 tx=0x7fb180008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.169+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb188054fb0 con 0x7fb1901599a0 2023-12-13T23:56:27.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.374+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb190069400 con 0x7fb1901599a0 2023-12-13T23:56:27.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.375+0000 7fb186ffd640 1 -- 172.21.15.124:0/3784865532 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4923 (secure 0 0 0) 0x7fb1880585d0 con 0x7fb1901599a0 2023-12-13T23:56:27.374 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:27.374 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:26.366668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:56:27.374 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 15 2023-12-13T23:56:27.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.376+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 msgr2=0x7fb17006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.376+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 0x7fb17006d680 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb180006fd0 tx=0x7fb180008040 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 msgr2=0x7fb19006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fb188007e60 tx=0x7fb18800e4b0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 shutdown_connections 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb17006b1c0 0x7fb17006d680 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb1901595c0 0x7fb1901eeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1901599a0 0x7fb19006d0b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 --2- 172.21.15.124:0/3784865532 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1900b3a50 0x7fb19014d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 >> 172.21.15.124:0/3784865532 conn(0x7fb190070e70 msgr2=0x7fb1900ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 shutdown_connections 2023-12-13T23:56:27.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.377+0000 7fb194d61640 1 -- 172.21.15.124:0/3784865532 wait complete. 2023-12-13T23:56:27.386 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:27.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.488+0000 7ff06f6e0640 1 Processor -- start 2023-12-13T23:56:27.487 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.488+0000 7ff06f6e0640 1 -- start start 2023-12-13T23:56:27.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06f6e0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff0680b1400 0x7ff06815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06f6e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06f6e0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06815e2a0 con 0x7ff068143490 2023-12-13T23:56:27.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06f6e0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06815b520 con 0x7ff0680b1400 2023-12-13T23:56:27.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06f6e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff068143870 con 0x7ff06815e730 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06dc7d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06dc7d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:49696/0 (socket says 172.21.15.124:49696) 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06cc7b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff0680b1400 0x7ff06815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.489+0000 7ff06dc7d640 1 -- 172.21.15.124:0/2402698731 learned_addr learned my addr 172.21.15.124:0/2402698731 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.490+0000 7ff06dc7d640 1 -- 172.21.15.124:0/2402698731 >> v2:172.21.15.203:3300/0 conn(0x7ff0680b1400 msgr2=0x7ff06815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.490+0000 7ff06dc7d640 1 --2- 172.21.15.124:0/2402698731 >> v2:172.21.15.203:3300/0 conn(0x7ff0680b1400 0x7ff06815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.490+0000 7ff06dc7d640 1 -- 172.21.15.124:0/2402698731 >> v1:172.21.15.203:6789/0 conn(0x7ff068143490 legacy=0x7ff0680b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.490+0000 7ff06dc7d640 1 -- 172.21.15.124:0/2402698731 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06815b240 con 0x7ff06815e730 2023-12-13T23:56:27.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06dc7d640 1 --2- 172.21.15.124:0/2402698731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff06000bb10 tx=0x7ff06002e150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbfa0e684baa71b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff05ffff640 1 -- 172.21.15.124:0/2402698731 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060036440 con 0x7ff06815e730 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff05ffff640 1 -- 172.21.15.124:0/2402698731 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0600365a0 con 0x7ff06815e730 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff05ffff640 1 -- 172.21.15.124:0/2402698731 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060036930 con 0x7ff06815e730 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 -- 172.21.15.124:0/2402698731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 msgr2=0x7ff06815aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/2402698731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff06000bb10 tx=0x7ff06002e150 comp rx=0 tx=0).stop 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 -- 172.21.15.124:0/2402698731 shutdown_connections 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/2402698731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815aa20 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/2402698731 >> v2:172.21.15.203:3300/0 conn(0x7ff0680b1400 0x7ff06815a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 -- 172.21.15.124:0/2402698731 >> 172.21.15.124:0/2402698731 conn(0x7ff06806a4d0 msgr2=0x7ff06806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 -- 172.21.15.124:0/2402698731 shutdown_connections 2023-12-13T23:56:27.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.491+0000 7ff06f6e0640 1 -- 172.21.15.124:0/2402698731 wait complete. 2023-12-13T23:56:27.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.492+0000 7ff06f6e0640 1 Processor -- start 2023-12-13T23:56:27.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.492+0000 7ff06f6e0640 1 -- start start 2023-12-13T23:56:27.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0680b1400 0x7ff06813dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06813f5f0 con 0x7ff068143490 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681eefc0 con 0x7ff0680b1400 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06f6e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681eecd0 con 0x7ff06815e730 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06cc7b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06cc7b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49586/0 (socket says 172.21.15.124:49586) 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06cc7b640 1 -- 172.21.15.124:0/3142185034 learned_addr learned my addr 172.21.15.124:0/3142185034 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06d47c640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0680b1400 0x7ff06813dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.493+0000 7ff06dc7d640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 -- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 msgr2=0x7ff06815c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 -- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0680b1400 msgr2=0x7ff06813dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0680b1400 0x7ff06813dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 -- 172.21.15.124:0/3142185034 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06000b770 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06dc7d640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06cc7b640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff05800e9b0 tx=0x7ff05800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058009670 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05800c9a0 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff06813ed50 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05800cd50 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.494+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06813f310 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.495+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff068069400 con 0x7ff068143490 2023-12-13T23:56:27.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.496+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff058013470 con 0x7ff068143490 2023-12-13T23:56:27.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.496+0000 7ff05dffb640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 0x7ff04c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.496+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff058089db0 con 0x7ff068143490 2023-12-13T23:56:27.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.496+0000 7ff06d47c640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 0x7ff04c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.497+0000 7ff06d47c640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 0x7ff04c06d680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff0640012e0 tx=0x7ff064009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.500+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff058054a00 con 0x7ff068143490 2023-12-13T23:56:27.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.667+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0680b11b0 con 0x7ff068143490 2023-12-13T23:56:27.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.668+0000 7ff05dffb640 1 -- 172.21.15.124:0/3142185034 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+18422 (secure 0 0 0) 0x7ff058017090 con 0x7ff068143490 2023-12-13T23:56:27.667 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:27.667 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":37,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:26.366456+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:27.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 msgr2=0x7ff04c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 0x7ff04c06d680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff0640012e0 tx=0x7ff064009040 comp rx=0 tx=0).stop 2023-12-13T23:56:27.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 msgr2=0x7ff0680a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff05800e9b0 tx=0x7ff05800ee70 comp rx=0 tx=0).stop 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 shutdown_connections 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff04c06b1c0 0x7ff04c06d680 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff06815e730 0x7ff06815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff068143490 0x7ff0680a84d0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 --2- 172.21.15.124:0/3142185034 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff0680b1400 0x7ff06813dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 >> 172.21.15.124:0/3142185034 conn(0x7ff06806a4d0 msgr2=0x7ff068141820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 shutdown_connections 2023-12-13T23:56:27.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.670+0000 7ff06f6e0640 1 -- 172.21.15.124:0/3142185034 wait complete. 2023-12-13T23:56:27.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:27.680 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2023-12-13T23:56:27.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:27.700 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2023-12-13T23:56:27.757 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2023-12-13T23:56:27.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.905+0000 7f58310c9640 1 Processor -- start 2023-12-13T23:56:27.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.905+0000 7f58310c9640 1 -- start start 2023-12-13T23:56:27.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f58310c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f582c13d050 0x7f582c0b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f58310c9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f582c0b1130 0x7f582c13faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f58310c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c0b1590 con 0x7f582c13ffe0 2023-12-13T23:56:27.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f58310c9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c13d430 con 0x7f582c0b1130 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f58310c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c15a680 con 0x7f582c13d050 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f582a59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f582c0b1130 0x7f582c13faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f582a59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f582c0b1130 0x7f582c13faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49588/0 (socket says 172.21.15.124:49588) 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f582a59c640 1 -- 172.21.15.124:0/2452082035 learned_addr learned my addr 172.21.15.124:0/2452082035 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f582ad9d640 1 --2- 172.21.15.124:0/2452082035 >> v2:172.21.15.203:3300/0 conn(0x7f582c13d050 0x7f582c0b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.906+0000 7f582a59c640 1 -- 172.21.15.124:0/2452082035 >> v2:172.21.15.203:3300/0 conn(0x7f582c13d050 msgr2=0x7f582c0b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f582a59c640 1 --2- 172.21.15.124:0/2452082035 >> v2:172.21.15.203:3300/0 conn(0x7f582c13d050 0x7f582c0b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f582a59c640 1 -- 172.21.15.124:0/2452082035 >> v1:172.21.15.203:6789/0 conn(0x7f582c13ffe0 legacy=0x7f582c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:27.906 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f582a59c640 1 -- 172.21.15.124:0/2452082035 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c15a940 con 0x7f582c0b1130 2023-12-13T23:56:27.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f582a59c640 1 --2- 172.21.15.124:0/2452082035 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c13faa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b1850 tx=0x7f582002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b17d8f1c8ea5276 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f5829d9b640 1 -- 172.21.15.124:0/2452082035 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820031870 con 0x7f582c0b1130 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f5829d9b640 1 -- 172.21.15.124:0/2452082035 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58200319d0 con 0x7f582c0b1130 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.907+0000 7f5829d9b640 1 -- 172.21.15.124:0/2452082035 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5820031d60 con 0x7f582c0b1130 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 -- 172.21.15.124:0/2452082035 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 msgr2=0x7f582c13faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 --2- 172.21.15.124:0/2452082035 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c13faa0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b1850 tx=0x7f582002b910 comp rx=0 tx=0).stop 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 -- 172.21.15.124:0/2452082035 shutdown_connections 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 --2- 172.21.15.124:0/2452082035 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c13faa0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 --2- 172.21.15.124:0/2452082035 >> v2:172.21.15.203:3300/0 conn(0x7f582c13d050 0x7f582c0b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 -- 172.21.15.124:0/2452082035 >> 172.21.15.124:0/2452082035 conn(0x7f582c06d9c0 msgr2=0x7f582c06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 -- 172.21.15.124:0/2452082035 shutdown_connections 2023-12-13T23:56:27.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.908+0000 7f58310c9640 1 -- 172.21.15.124:0/2452082035 wait complete. 2023-12-13T23:56:27.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.909+0000 7f58310c9640 1 Processor -- start 2023-12-13T23:56:27.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.909+0000 7f58310c9640 1 -- start start 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c13d050 0x7f582c071750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c13ffe0 0x7f582c15c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c156c70 con 0x7f582c0b1130 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c157090 con 0x7f582c13d050 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f58310c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c155ff0 con 0x7f582c13ffe0 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49600/0 (socket says 172.21.15.124:49600) 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 -- 172.21.15.124:0/3374117323 learned_addr learned my addr 172.21.15.124:0/3374117323 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 -- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c13ffe0 msgr2=0x7f582c15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c13ffe0 0x7f582c15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 -- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c13d050 msgr2=0x7f582c071750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:27.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c13d050 0x7f582c071750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.910+0000 7f582ad9d640 1 -- 172.21.15.124:0/3374117323 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58200094d0 con 0x7f582c0b1130 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f582ad9d640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f581c007ef0 tx=0x7f581c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c014620 con 0x7f582c0b1130 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c014780 con 0x7f582c0b1130 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c1562d0 con 0x7f582c0b1130 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c014b30 con 0x7f582c0b1130 2023-12-13T23:56:27.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.911+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c156890 con 0x7f582c0b1130 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.912+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c069400 con 0x7f582c0b1130 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.912+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f581c014c90 con 0x7f582c0b1130 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.913+0000 7f58137fe640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 0x7f581806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.913+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f581c08a500 con 0x7f582c0b1130 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.913+0000 7f582a59c640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 0x7f581806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:27.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.913+0000 7f582a59c640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 0x7f581806d680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5820036000 tx=0x7f5820030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:27.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:27.917+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f581c055130 con 0x7f582c0b1130 2023-12-13T23:56:28.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.151+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7f582c06a780 con 0x7f582c0b1130 2023-12-13T23:56:28.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.153+0000 7f58137fe640 1 -- 172.21.15.124:0/3374117323 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v12) v1 ==== 210+0+167 (secure 0 0 0) 0x7f581c058750 con 0x7f582c0b1130 2023-12-13T23:56:28.151 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:56:28.152 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:56:28.152 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow rw" 2023-12-13T23:56:28.152 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:56:28.152 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2023-12-13T23:56:28.152 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.154+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 msgr2=0x7f581806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.154+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 0x7f581806d680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5820036000 tx=0x7f5820030000 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.154+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 msgr2=0x7f582c137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.154+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f581c007ef0 tx=0x7f581c00e4b0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 shutdown_connections 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581806b1c0 0x7f581806d680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c13d050 0x7f582c071750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c13ffe0 0x7f582c15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 --2- 172.21.15.124:0/3374117323 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0b1130 0x7f582c137820 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 >> 172.21.15.124:0/3374117323 conn(0x7f582c06d9c0 msgr2=0x7f582c158c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 shutdown_connections 2023-12-13T23:56:28.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.155+0000 7f58310c9640 1 -- 172.21.15.124:0/3374117323 wait complete. 2023-12-13T23:56:28.164 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:28.164 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:28.179 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:28.318 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.320+0000 7f7e53689640 1 Processor -- start 2023-12-13T23:56:28.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.320+0000 7f7e53689640 1 -- start start 2023-12-13T23:56:28.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e53689640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7e4c15d1d0 0x7f7e4c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e53689640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e4c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e53689640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c0adae0 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e53689640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0addc0 con 0x7f7e4c15d1d0 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e53689640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0ae0a0 con 0x7f7e4c0ad640 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e51c26640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f7e4c0aeb80 0x7f7e4c160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36262/0 (socket says 172.21.15.124:36262) 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e51c26640 1 -- 172.21.15.124:0/1334476382 learned_addr learned my addr 172.21.15.124:0/1334476382 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269504313 0 0) 0x7f7e4c0adae0 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e51425640 1 --2- 172.21.15.124:0/1334476382 >> v2:172.21.15.203:3300/0 conn(0x7f7e4c15d1d0 0x7f7e4c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e38003520 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e50c24640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e4c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3624801413 0 0) 0x7f7e38003520 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 msgr2=0x7f7e4c06ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e4c06ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 >> v2:172.21.15.203:3300/0 conn(0x7f7e4c15d1d0 msgr2=0x7f7e4c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 --2- 172.21.15.124:0/1334476382 >> v2:172.21.15.203:3300/0 conn(0x7f7e4c15d1d0 0x7f7e4c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c0ae380 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f7e440019f0 con 0x7f7e4c0aeb80 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 >> v1:172.21.15.124:6789/0 conn(0x7f7e4c0aeb80 legacy=0x7f7e4c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e38003970 0x7f7e38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38004d00 0x7f7e38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0addc0 con 0x7f7e38003970 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c0ae0a0 con 0x7f7e38004d00 2023-12-13T23:56:28.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e3800b980 con 0x7f7e4c0ad640 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.321+0000 7f7e51425640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e38003970 0x7f7e38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e50c24640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51c26640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38004d00 0x7f7e38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 -- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 msgr2=0x7f7e380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e4c0ad640 0x7f7e380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 -- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38004d00 msgr2=0x7f7e38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38004d00 0x7f7e38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 -- 172.21.15.124:0/1334476382 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38003520 con 0x7f7e38003970 2023-12-13T23:56:28.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51c26640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e38004d00 0x7f7e38005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e51425640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e38003970 0x7f7e38003d90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009ac0 tx=0x7f7e380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a8a8206ec995d7 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4802c710 con 0x7f7e38003970 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4802c870 con 0x7f7e38003970 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.322+0000 7f7e43fff640 1 -- 172.21.15.124:0/1334476382 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4802cc00 con 0x7f7e38003970 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 -- 172.21.15.124:0/1334476382 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e38003970 msgr2=0x7f7e38003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 --2- 172.21.15.124:0/1334476382 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e38003970 0x7f7e38003d90 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009ac0 tx=0x7f7e380057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e51c26640 1 -- 172.21.15.124:0/1334476382 reap_dead start 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 -- 172.21.15.124:0/1334476382 shutdown_connections 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 -- 172.21.15.124:0/1334476382 >> 172.21.15.124:0/1334476382 conn(0x7f7e4c070e70 msgr2=0x7f7e4c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 -- 172.21.15.124:0/1334476382 shutdown_connections 2023-12-13T23:56:28.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.323+0000 7f7e53689640 1 -- 172.21.15.124:0/1334476382 wait complete. 2023-12-13T23:56:28.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.324+0000 7f7e53689640 1 Processor -- start 2023-12-13T23:56:28.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.324+0000 7f7e53689640 1 -- start start 2023-12-13T23:56:28.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e38003970 0x7f7e4c1f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e4c160470 0x7f7e4c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c152a70 con 0x7f7e4c15fff0 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c1f3d60 con 0x7f7e4c160470 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e53689640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c158ed0 con 0x7f7e38003970 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49610/0 (socket says 172.21.15.124:49610) 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 -- 172.21.15.124:0/3923811082 learned_addr learned my addr 172.21.15.124:0/3923811082 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e51c26640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e4c160470 0x7f7e4c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e51425640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e38003970 0x7f7e4c1f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 -- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e38003970 msgr2=0x7f7e4c1f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e38003970 0x7f7e4c1f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 -- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e4c160470 msgr2=0x7f7e4c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e4c160470 0x7f7e4c152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.325+0000 7f7e50c24640 1 -- 172.21.15.124:0/3923811082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c159170 con 0x7f7e4c15fff0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e50c24640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0119a0 tx=0x7f7e3c011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c00fa30 con 0x7f7e4c15fff0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3c00fb90 con 0x7f7e4c15fff0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c159450 con 0x7f7e4c15fff0 2023-12-13T23:56:28.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3c014440 con 0x7f7e4c15fff0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.326+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c159a10 con 0x7f7e4c15fff0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.327+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c069400 con 0x7f7e4c15fff0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.327+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7e3c0145a0 con 0x7f7e4c15fff0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.328+0000 7f7e41ffb640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 0x7f7e3006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.328+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7e3c08a470 con 0x7f7e4c15fff0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.328+0000 7f7e51425640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 0x7f7e3006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.328+0000 7f7e51425640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 0x7f7e3006d680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.332+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7e3c055010 con 0x7f7e4c15fff0 2023-12-13T23:56:28.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.499+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e4c0b2ae0 con 0x7f7e4c15fff0 2023-12-13T23:56:28.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.501+0000 7f7e41ffb640 1 -- 172.21.15.124:0/3923811082 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f7e3c058630 con 0x7f7e4c15fff0 2023-12-13T23:56:28.500 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:28.500 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:28.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 msgr2=0x7f7e3006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 0x7f7e3006d680 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7e48036000 tx=0x7f7e48030000 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 msgr2=0x7f7e4c151ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f7e3c0119a0 tx=0x7f7e3c011da0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 shutdown_connections 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7e3006b1c0 0x7f7e3006d680 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7e4c160470 0x7f7e4c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7e4c15fff0 0x7f7e4c151ff0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 --2- 172.21.15.124:0/3923811082 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7e38003970 0x7f7e4c1f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.502+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 >> 172.21.15.124:0/3923811082 conn(0x7f7e4c070e70 msgr2=0x7f7e4c154ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.503+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 shutdown_connections 2023-12-13T23:56:28.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.503+0000 7f7e53689640 1 -- 172.21.15.124:0/3923811082 wait complete. 2023-12-13T23:56:28.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:28.513 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:28.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 Processor -- start 2023-12-13T23:56:28.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 -- start start 2023-12-13T23:56:28.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdfa015d330 0x7fdfa00b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa00d3900 0x7fdfa00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfa00b2b10 con 0x7fdfa00d2910 2023-12-13T23:56:28.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa00d1590 con 0x7fdfa015d330 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.616+0000 7fdfa5197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00d1870 con 0x7fdfa00d3900 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdfa015d330 0x7fdfa00b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdfa015d330 0x7fdfa00b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49622/0 (socket says 172.21.15.124:49622) 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 -- 172.21.15.124:0/3225674844 learned_addr learned my addr 172.21.15.124:0/3225674844 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9f59e640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa00d3900 0x7fdfa00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 -- 172.21.15.124:0/3225674844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa00d3900 msgr2=0x7fdfa00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa00d3900 0x7fdfa00b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 -- 172.21.15.124:0/3225674844 >> v1:172.21.15.203:6789/0 conn(0x7fdfa00d2910 legacy=0x7fdfa00d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.617+0000 7fdf9ed9d640 1 -- 172.21.15.124:0/3225674844 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa00d1b50 con 0x7fdfa015d330 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdf9ed9d640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa015d330 0x7fdfa00b8b80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9c7ba8f687122e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdf9e59c640 1 -- 172.21.15.124:0/3225674844 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90031640 con 0x7fdfa015d330 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdf9e59c640 1 -- 172.21.15.124:0/3225674844 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf900317a0 con 0x7fdfa015d330 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdf9e59c640 1 -- 172.21.15.124:0/3225674844 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf90031b30 con 0x7fdfa015d330 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 -- 172.21.15.124:0/3225674844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa015d330 msgr2=0x7fdfa00b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa015d330 0x7fdfa00b8b80 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdf90009870 tx=0x7fdf9002b8f0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 -- 172.21.15.124:0/3225674844 shutdown_connections 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa00d3900 0x7fdfa00b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3225674844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa015d330 0x7fdfa00b8b80 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 -- 172.21.15.124:0/3225674844 >> 172.21.15.124:0/3225674844 conn(0x7fdfa0072020 msgr2=0x7fdfa0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 -- 172.21.15.124:0/3225674844 shutdown_connections 2023-12-13T23:56:28.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.618+0000 7fdfa5197640 1 -- 172.21.15.124:0/3225674844 wait complete. 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 Processor -- start 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 -- start start 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa00d2910 0x7fdfa0152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa015d330 0x7fdfa0154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c8bb0 con 0x7fdfa00d3900 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0154570 con 0x7fdfa00d2910 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.620+0000 7fdfa5197640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00c82c0 con 0x7fdfa015d330 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49636/0 (socket says 172.21.15.124:49636) 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 -- 172.21.15.124:0/3762801460 learned_addr learned my addr 172.21.15.124:0/3762801460 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf9ed9d640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa00d2910 0x7fdfa0152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf9f59e640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa015d330 0x7fdfa0154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 -- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa015d330 msgr2=0x7fdfa0154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa015d330 0x7fdfa0154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 -- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa00d2910 msgr2=0x7fdfa0152910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa00d2910 0x7fdfa0152910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 -- 172.21.15.124:0/3762801460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf900094d0 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf97fff640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800c400 tx=0x7fdf8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf880052d0 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88005430 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa00c8560 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88003830 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.621+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa01547d0 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.622+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa0069400 con 0x7fdfa00d3900 2023-12-13T23:56:28.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.623+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdf88003990 con 0x7fdfa00d3900 2023-12-13T23:56:28.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.623+0000 7fdf977fe640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 0x7fdf8006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.623+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fdf88088f90 con 0x7fdfa00d3900 2023-12-13T23:56:28.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.623+0000 7fdf9ed9d640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 0x7fdf8006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.624+0000 7fdf9ed9d640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 0x7fdf8006d6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036000 tx=0x7fdf9003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.627+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdf880bbe90 con 0x7fdfa00d3900 2023-12-13T23:56:28.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.793+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdfa00725a0 con 0x7fdfa00d3900 2023-12-13T23:56:28.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.793+0000 7fdf977fe640 1 -- 172.21.15.124:0/3762801460 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf88017020 con 0x7fdfa00d3900 2023-12-13T23:56:28.792 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 msgr2=0x7fdf8006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 0x7fdf8006d6b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf90036000 tx=0x7fdf9003c040 comp rx=0 tx=0).stop 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 msgr2=0x7fdfa01f6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdf8800c400 tx=0x7fdf8800c8c0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 shutdown_connections 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdf8006b1f0 0x7fdf8006d6b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdfa015d330 0x7fdfa0154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfa00d3900 0x7fdfa01f6da0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 --2- 172.21.15.124:0/3762801460 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdfa00d2910 0x7fdfa0152910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 >> 172.21.15.124:0/3762801460 conn(0x7fdfa0072020 msgr2=0x7fdfa00b6350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 shutdown_connections 2023-12-13T23:56:28.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.795+0000 7fdfa5197640 1 -- 172.21.15.124:0/3762801460 wait complete. 2023-12-13T23:56:28.805 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:28.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.908+0000 7ffba4eb4640 1 Processor -- start 2023-12-13T23:56:28.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.908+0000 7ffba4eb4640 1 -- start start 2023-12-13T23:56:28.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffba4eb4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffba4eb4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffba4eb4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba0071770 con 0x7ffba00ceb70 2023-12-13T23:56:28.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffba4eb4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba00ccb60 con 0x7ffba0144980 2023-12-13T23:56:28.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffba4eb4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00b4ce0 con 0x7ffba00b4370 2023-12-13T23:56:28.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9ed9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9e59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9ed9d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:49754/0 (socket says 172.21.15.124:49754) 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9dd9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ffba00ceb70 0x7ffba00cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:54688/0 (socket says 172.21.15.124:54688) 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9ed9d640 1 -- 172.21.15.124:0/3748512193 learned_addr learned my addr 172.21.15.124:0/3748512193 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.909+0000 7ffb9e59c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:32824/0 (socket says 172.21.15.124:32824) 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9ed9d640 1 -- 172.21.15.124:0/3748512193 >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 msgr2=0x7ffba015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9ed9d640 1 --2- 172.21.15.124:0/3748512193 >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9ed9d640 1 -- 172.21.15.124:0/3748512193 >> v1:172.21.15.203:6789/0 conn(0x7ffba00ceb70 legacy=0x7ffba00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9ed9d640 1 -- 172.21.15.124:0/3748512193 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba00ccdc0 con 0x7ffba00b4370 2023-12-13T23:56:28.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9e59c640 1 --2- 172.21.15.124:0/3748512193 >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:28.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.910+0000 7ffb9ed9d640 1 --2- 172.21.15.124:0/3748512193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009e50 tx=0x7ffb9802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff6ed3090c0ab565 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffb9d59a640 1 -- 172.21.15.124:0/3748512193 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9802cb00 con 0x7ffba00b4370 2023-12-13T23:56:28.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffb9d59a640 1 -- 172.21.15.124:0/3748512193 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9802cc60 con 0x7ffba00b4370 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffb9d59a640 1 -- 172.21.15.124:0/3748512193 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb980343f0 con 0x7ffba00b4370 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 -- 172.21.15.124:0/3748512193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 msgr2=0x7ffba00b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/3748512193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ffb98009e50 tx=0x7ffb9802bf60 comp rx=0 tx=0).stop 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 -- 172.21.15.124:0/3748512193 shutdown_connections 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/3748512193 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00b4370 0x7ffba00b47a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/3748512193 >> v2:172.21.15.203:3300/0 conn(0x7ffba0144980 0x7ffba015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 -- 172.21.15.124:0/3748512193 >> 172.21.15.124:0/3748512193 conn(0x7ffba006a4d0 msgr2=0x7ffba006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 -- 172.21.15.124:0/3748512193 shutdown_connections 2023-12-13T23:56:28.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.911+0000 7ffba4eb4640 1 -- 172.21.15.124:0/3748512193 wait complete. 2023-12-13T23:56:28.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.912+0000 7ffba4eb4640 1 Processor -- start 2023-12-13T23:56:28.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.912+0000 7ffba4eb4640 1 -- start start 2023-12-13T23:56:28.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 0x7ffba01339e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 0x7ffba00c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba00c9230 con 0x7ffba00b4370 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba01553d0 con 0x7ffba0144980 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffba4eb4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba0131780 con 0x7ffba00ceb70 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffb9e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffb9e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49642/0 (socket says 172.21.15.124:49642) 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.913+0000 7ffb9e59c640 1 -- 172.21.15.124:0/593804381 learned_addr learned my addr 172.21.15.124:0/593804381 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9ed9d640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 0x7ffba00c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9dd9b640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 0x7ffba01339e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 -- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 msgr2=0x7ffba01339e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 0x7ffba01339e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 -- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 msgr2=0x7ffba00c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 0x7ffba00c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:28.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 -- 172.21.15.124:0/593804381 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb90009400 con 0x7ffba00b4370 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9ed9d640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 0x7ffba00c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9dd9b640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 0x7ffba01339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb9e59c640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000e960 tx=0x7ffb9000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9000ca00 con 0x7ffba00b4370 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb9000cb60 con 0x7ffba00b4370 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb98009b30 con 0x7ffba00b4370 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90013470 con 0x7ffba00b4370 2023-12-13T23:56:28.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.914+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba0132020 con 0x7ffba00b4370 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.915+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba0069400 con 0x7ffba00b4370 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.915+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ffb900135d0 con 0x7ffba00b4370 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.916+0000 7ffb8effd640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 0x7ffb8006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.916+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ffb90089e50 con 0x7ffba00b4370 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.916+0000 7ffb9dd9b640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 0x7ffb8006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:28.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.917+0000 7ffb9dd9b640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 0x7ffb8006d680 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb94007fd0 tx=0x7ffb94007d40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:28.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:28.920+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ffb900549f0 con 0x7ffba00b4370 2023-12-13T23:56:29.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.089+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffba015e7c0 con 0x7ffba00b4370 2023-12-13T23:56:29.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.089+0000 7ffb8effd640 1 -- 172.21.15.124:0/593804381 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffb90058010 con 0x7ffba00b4370 2023-12-13T23:56:29.088 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:29.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 msgr2=0x7ffb8006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 0x7ffb8006d680 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ffb94007fd0 tx=0x7ffb94007d40 comp rx=0 tx=0).stop 2023-12-13T23:56:29.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 msgr2=0x7ffba00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000e960 tx=0x7ffb9000ee20 comp rx=0 tx=0).stop 2023-12-13T23:56:29.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 shutdown_connections 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffb8006b1c0 0x7ffb8006d680 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffba0144980 0x7ffba00c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffba00ceb70 0x7ffba01339e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 --2- 172.21.15.124:0/593804381 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffba00b4370 0x7ffba00c8cf0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 >> 172.21.15.124:0/593804381 conn(0x7ffba006a4d0 msgr2=0x7ffba01411f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 shutdown_connections 2023-12-13T23:56:29.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.091+0000 7ffba4eb4640 1 -- 172.21.15.124:0/593804381 wait complete. 2023-12-13T23:56:29.101 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:29.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.204+0000 7f01aed64640 1 Processor -- start 2023-12-13T23:56:29.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01aed64640 1 -- start start 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01aed64640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01a80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01aed64640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80b2b10 con 0x7f01a80d2d90 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01aed64640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80d1590 con 0x7f01a80d2910 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01aed64640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80d1870 con 0x7f01a815d330 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01ad301640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f01a80d2d90 0x7f01a80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36264/0 (socket says 172.21.15.124:36264) 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.205+0000 7f01ad301640 1 -- 172.21.15.124:0/3433317368 learned_addr learned my addr 172.21.15.124:0/3433317368 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01acb00640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01a80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870452506 0 0) 0x7f01a80b2b10 con 0x7f01a80d2d90 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0190003520 con 0x7f01a80d2d90 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425241941 0 0) 0x7f01a80d1590 con 0x7f01a80d2910 2023-12-13T23:56:29.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a80b2b10 con 0x7f01a80d2910 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371654249 0 0) 0x7f0190003520 con 0x7f01a80d2d90 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 msgr2=0x7f01a80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01a80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 >> v1:172.21.15.203:6789/0 conn(0x7f01a80d2910 legacy=0x7f01a80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80d1b50 con 0x7f01a80d2d90 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f01a00019f0 con 0x7f01a80d2d90 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 >> v1:172.21.15.124:6789/0 conn(0x7f01a80d2d90 legacy=0x7f01a80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0190003b30 0x7f0190003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0190004d00 0x7f0190005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80d1870 con 0x7f0190004d00 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019000ba80 con 0x7f0190003b30 2023-12-13T23:56:29.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019000bc40 con 0x7f01a815d330 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0190004d00 0x7f0190005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01acb00640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0190003b30 0x7f0190003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01a7fff640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 -- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 msgr2=0x7f01900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815d330 0x7f01900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 -- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0190003b30 msgr2=0x7f0190003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0190003b30 0x7f0190003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.206+0000 7f01ad301640 1 -- 172.21.15.124:0/3433317368 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0190003520 con 0x7f0190004d00 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.207+0000 7f01ad301640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0190004d00 0x7f0190005180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f01a80ca2e0 tx=0x7f01900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2be1d49abf133478 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.207+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a0034630 con 0x7f0190004d00 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.207+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a002f070 con 0x7f0190004d00 2023-12-13T23:56:29.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.207+0000 7f01a77fe640 1 -- 172.21.15.124:0/3433317368 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a00349f0 con 0x7f0190004d00 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 -- 172.21.15.124:0/3433317368 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0190004d00 msgr2=0x7f0190005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 --2- 172.21.15.124:0/3433317368 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0190004d00 0x7f0190005180 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f01a80ca2e0 tx=0x7f01900057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01ad301640 1 -- 172.21.15.124:0/3433317368 reap_dead start 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 -- 172.21.15.124:0/3433317368 shutdown_connections 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 -- 172.21.15.124:0/3433317368 >> 172.21.15.124:0/3433317368 conn(0x7f01a8072020 msgr2=0x7f01a8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 -- 172.21.15.124:0/3433317368 shutdown_connections 2023-12-13T23:56:29.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.208+0000 7f01aed64640 1 -- 172.21.15.124:0/3433317368 wait complete. 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.209+0000 7f01aed64640 1 Processor -- start 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.209+0000 7f01aed64640 1 -- start start 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 0x7f01a80ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a80cc1f0 0x7f01a815b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80cac10 con 0x7f01a80c8a10 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80caef0 con 0x7f01a80cc1f0 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01aed64640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a80cb1d0 con 0x7f01a8158e30 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49654/0 (socket says 172.21.15.124:49654) 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 -- 172.21.15.124:0/4199958993 learned_addr learned my addr 172.21.15.124:0/4199958993 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01ad301640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a80cc1f0 0x7f01a815b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01acb00640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 0x7f01a80ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 -- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 msgr2=0x7f01a80ca330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 0x7f01a80ca330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 -- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a80cc1f0 msgr2=0x7f01a815b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a80cc1f0 0x7f01a815b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 -- 172.21.15.124:0/4199958993 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80cb4b0 con 0x7f01a80c8a10 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01acb00640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 0x7f01a80ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a7fff640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f019c00f7a0 tx=0x7f019c00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.210+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00fe50 con 0x7f01a80c8a10 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.211+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c0052e0 con 0x7f01a80c8a10 2023-12-13T23:56:29.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.211+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c00d970 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.211+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a80cb790 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.211+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a8159210 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.212+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80cb9b0 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.212+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f019c00dad0 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.212+0000 7f01a57fa640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 0x7f019406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.213+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f019c089b30 con 0x7f01a80c8a10 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.213+0000 7f01acb00640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 0x7f019406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.213+0000 7f01acb00640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 0x7f019406d680 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f01a81f65f0 tx=0x7f0198008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.217+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f019c054760 con 0x7f01a80c8a10 2023-12-13T23:56:29.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.384+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f01a8069400 con 0x7f01a80c8a10 2023-12-13T23:56:29.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.384+0000 7f01a57fa640 1 -- 172.21.15.124:0/4199958993 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7f019c057d80 con 0x7f01a80c8a10 2023-12-13T23:56:29.383 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 msgr2=0x7f019406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 0x7f019406d680 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f01a81f65f0 tx=0x7f0198008040 comp rx=0 tx=0).stop 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 msgr2=0x7f01a80cbcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f019c00f7a0 tx=0x7f019c00fc60 comp rx=0 tx=0).stop 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 shutdown_connections 2023-12-13T23:56:29.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f019406b1c0 0x7f019406d680 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a80cc1f0 0x7f01a815b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a80c8a10 0x7f01a80cbcb0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 --2- 172.21.15.124:0/4199958993 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8158e30 0x7f01a80ca330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 >> 172.21.15.124:0/4199958993 conn(0x7f01a8072020 msgr2=0x7f01a80b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 shutdown_connections 2023-12-13T23:56:29.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.386+0000 7f01aed64640 1 -- 172.21.15.124:0/4199958993 wait complete. 2023-12-13T23:56:29.395 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:29.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.501+0000 7f492cde3640 1 Processor -- start 2023-12-13T23:56:29.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.501+0000 7f492cde3640 1 -- start start 2023-12-13T23:56:29.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492cde3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f49280b4370 0x7f49280b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492cde3640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4928071770 con 0x7f4928144980 2023-12-13T23:56:29.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492cde3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280ccb60 con 0x7f49280ceb70 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492cde3640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f492815ced0 con 0x7f49280b4370 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492659c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f4928144980 0x7f492815c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36272/0 (socket says 172.21.15.124:36272) 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492659c640 1 -- 172.21.15.124:0/2331496126 learned_addr learned my addr 172.21.15.124:0/2331496126 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f4926d9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f49280b4370 0x7f49280b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f4926d9d640 1 -- 172.21.15.124:0/2331496126 >> v1:172.21.15.203:6789/0 conn(0x7f49280ceb70 legacy=0x7f49280cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f4926d9d640 1 -- 172.21.15.124:0/2331496126 >> v1:172.21.15.124:6789/0 conn(0x7f4928144980 legacy=0x7f492815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f492559a640 1 -- 172.21.15.124:0/2331496126 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738860787 0 0) 0x7f4928071770 con 0x7f4928144980 2023-12-13T23:56:29.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.502+0000 7f4926d9d640 1 -- 172.21.15.124:0/2331496126 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280b4ce0 con 0x7f49280b4370 2023-12-13T23:56:29.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.503+0000 7f4926d9d640 1 --2- 172.21.15.124:0/2331496126 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49280b4370 0x7f49280b47a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f49280b4e70 tx=0x7f492002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30bd6f5506bb62dd server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.503+0000 7f492559a640 1 -- 172.21.15.124:0/2331496126 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920030610 con 0x7f49280b4370 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.503+0000 7f492559a640 1 -- 172.21.15.124:0/2331496126 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920030770 con 0x7f49280b4370 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.503+0000 7f492559a640 1 -- 172.21.15.124:0/2331496126 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920030b00 con 0x7f49280b4370 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 -- 172.21.15.124:0/2331496126 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49280b4370 msgr2=0x7f49280b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 --2- 172.21.15.124:0/2331496126 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49280b4370 0x7f49280b47a0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f49280b4e70 tx=0x7f492002b960 comp rx=0 tx=0).stop 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 -- 172.21.15.124:0/2331496126 shutdown_connections 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 --2- 172.21.15.124:0/2331496126 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49280b4370 0x7f49280b47a0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 -- 172.21.15.124:0/2331496126 >> 172.21.15.124:0/2331496126 conn(0x7f492806a4d0 msgr2=0x7f492806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 -- 172.21.15.124:0/2331496126 shutdown_connections 2023-12-13T23:56:29.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.504+0000 7f492cde3640 1 -- 172.21.15.124:0/2331496126 wait complete. 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.505+0000 7f492cde3640 1 Processor -- start 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.505+0000 7f492cde3640 1 -- start start 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280b4370 0x7f492815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49280ceb70 0x7f492815b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49281f43a0 con 0x7f4928144980 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49281f4680 con 0x7f49280ceb70 2023-12-13T23:56:29.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492cde3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49281f4960 con 0x7f49280b4370 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49670/0 (socket says 172.21.15.124:49670) 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 -- 172.21.15.124:0/2350820817 learned_addr learned my addr 172.21.15.124:0/2350820817 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f492659c640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280b4370 0x7f492815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4925d9b640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49280ceb70 0x7f492815b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 -- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280b4370 msgr2=0x7f492815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280b4370 0x7f492815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 -- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49280ceb70 msgr2=0x7f492815b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49280ceb70 0x7f492815b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 -- 172.21.15.124:0/2350820817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4920009400 con 0x7f4928144980 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.506+0000 7f4926d9d640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4920009890 tx=0x7f4920004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.507+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492003e040 con 0x7f4928144980 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.507+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920039c10 con 0x7f4928144980 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.507+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49281f4c40 con 0x7f4928144980 2023-12-13T23:56:29.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.507+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49200443f0 con 0x7f4928144980 2023-12-13T23:56:29.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.507+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49281f5200 con 0x7f4928144980 2023-12-13T23:56:29.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.508+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928069400 con 0x7f4928144980 2023-12-13T23:56:29.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.508+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f492004d3b0 con 0x7f4928144980 2023-12-13T23:56:29.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.508+0000 7f4916ffd640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 0x7f490806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.509+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f49200ac270 con 0x7f4928144980 2023-12-13T23:56:29.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.509+0000 7f492659c640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 0x7f490806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.509+0000 7f492659c640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 0x7f490806d680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f49180012e0 tx=0x7f4918005c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.513+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4920076e10 con 0x7f4928144980 2023-12-13T23:56:29.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.678+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f4928155400 con 0x7f4928144980 2023-12-13T23:56:29.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.678+0000 7f4916ffd640 1 -- 172.21.15.124:0/2350820817 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f492007a430 con 0x7f4928144980 2023-12-13T23:56:29.677 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:29.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 msgr2=0x7f490806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 0x7f490806d680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f49180012e0 tx=0x7f4918005c10 comp rx=0 tx=0).stop 2023-12-13T23:56:29.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 msgr2=0x7f492813f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4920009890 tx=0x7f4920004730 comp rx=0 tx=0).stop 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 shutdown_connections 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f490806b1c0 0x7f490806d680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49280ceb70 0x7f492815b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928144980 0x7f492813f450 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 --2- 172.21.15.124:0/2350820817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280b4370 0x7f492815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 >> 172.21.15.124:0/2350820817 conn(0x7f492806a4d0 msgr2=0x7f492815c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 shutdown_connections 2023-12-13T23:56:29.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.680+0000 7f492cde3640 1 -- 172.21.15.124:0/2350820817 wait complete. 2023-12-13T23:56:29.690 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:29.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.792+0000 7f18ba67c640 1 Processor -- start 2023-12-13T23:56:29.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.792+0000 7f18ba67c640 1 -- start start 2023-12-13T23:56:29.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18ba67c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f18b415d2b0 0x7f18b4071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18ba67c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b40ae910 con 0x7f18b4072840 2023-12-13T23:56:29.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18ba67c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b40b2200 con 0x7f18b40ae480 2023-12-13T23:56:29.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18ba67c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b40b24e0 con 0x7f18b415d2b0 2023-12-13T23:56:29.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b8c19640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f18b4072840 0x7f18b4168000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36288/0 (socket says 172.21.15.124:36288) 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b8c19640 1 -- 172.21.15.124:0/2071650017 learned_addr learned my addr 172.21.15.124:0/2071650017 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b3fff640 1 --2- 172.21.15.124:0/2071650017 >> v2:172.21.15.203:3300/0 conn(0x7f18b415d2b0 0x7f18b4071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938450811 0 0) 0x7f18b40ae910 con 0x7f18b4072840 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f189c003520 con 0x7f18b4072840 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.793+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 694542956 0 0) 0x7f189c003520 con 0x7f18b4072840 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 >> v2:172.21.15.203:3300/0 conn(0x7f18b415d2b0 msgr2=0x7f18b4071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 --2- 172.21.15.124:0/2071650017 >> v2:172.21.15.203:3300/0 conn(0x7f18b415d2b0 0x7f18b4071c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 >> v1:172.21.15.203:6789/0 conn(0x7f18b40ae480 legacy=0x7f18b406d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40b40a0 con 0x7f18b4072840 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f18ac0019f0 con 0x7f18b4072840 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 >> v1:172.21.15.124:6789/0 conn(0x7f18b4072840 legacy=0x7f18b4168000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c003970 0x7f189c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 0x7f189c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f189c005180 0x7f189c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b40b24e0 con 0x7f189c0047c0 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189c00c120 con 0x7f189c005180 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189c00c2e0 con 0x7f189c003970 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 0x7f189c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b8c19640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f189c005180 0x7f189c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b3fff640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c003970 0x7f189c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 -- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c003970 msgr2=0x7f189c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c003970 0x7f189c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 -- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f189c005180 msgr2=0x7f189c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f189c005180 0x7f189c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b37fe640 1 -- 172.21.15.124:0/2071650017 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f189c003520 con 0x7f189c0047c0 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b8c19640 1 -- 172.21.15.124:0/2071650017 reap_dead start 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.794+0000 7f18b3fff640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c003970 0x7f189c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:29.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18b37fe640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 0x7f189c004c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f18a8001750 tx=0x7f189c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bff9dfcedbc470f2 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a80052d0 con 0x7f189c0047c0 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a802e050 con 0x7f189c0047c0 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18b2ffd640 1 -- 172.21.15.124:0/2071650017 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8003830 con 0x7f189c0047c0 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 -- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 msgr2=0x7f189c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 0x7f189c004c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f18a8001750 tx=0x7f189c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 -- 172.21.15.124:0/2071650017 shutdown_connections 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 --2- 172.21.15.124:0/2071650017 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f189c0047c0 0x7f189c004c40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 -- 172.21.15.124:0/2071650017 >> 172.21.15.124:0/2071650017 conn(0x7f18b40b4720 msgr2=0x7f18b40b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 -- 172.21.15.124:0/2071650017 shutdown_connections 2023-12-13T23:56:29.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.795+0000 7f18ba67c640 1 -- 172.21.15.124:0/2071650017 wait complete. 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 Processor -- start 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 -- start start 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c0047c0 0x7f18b414d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18b415a9c0 0x7f18b415ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b41f2a50 con 0x7f18b4151980 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b415b3b0 con 0x7f18b415a9c0 2023-12-13T23:56:29.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.797+0000 7f18ba67c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b4159920 con 0x7f189c0047c0 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49696/0 (socket says 172.21.15.124:49696) 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 -- 172.21.15.124:0/79455039 learned_addr learned my addr 172.21.15.124:0/79455039 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b8c19640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18b415a9c0 0x7f18b415ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b3fff640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c0047c0 0x7f18b414d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 -- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c0047c0 msgr2=0x7f18b414d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c0047c0 0x7f18b414d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 -- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18b415a9c0 msgr2=0x7f18b415ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18b415a9c0 0x7f18b415ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 -- 172.21.15.124:0/79455039 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b40b2af0 con 0x7f18b4151980 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b37fe640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f18a8005040 tx=0x7f18a8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8007a30 con 0x7f18b4151980 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18a80069e0 con 0x7f18b4151980 2023-12-13T23:56:29.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b4159c00 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18a8006d70 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.798+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b415a1c0 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.799+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b4069400 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.800+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f18a80035f0 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.800+0000 7f18b0ff9640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 0x7f18a006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.800+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f18a80abc50 con 0x7f18b4151980 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.800+0000 7f18b3fff640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 0x7f18a006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:29.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.801+0000 7f18b3fff640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 0x7f18a006d6b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18b4050020 tx=0x7f18a4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:29.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.804+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f18a80b1020 con 0x7f18b4151980 2023-12-13T23:56:29.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.981+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f18b4160480 con 0x7f18b4151980 2023-12-13T23:56:29.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.981+0000 7f18b0ff9640 1 -- 172.21.15.124:0/79455039 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7f18a804b020 con 0x7f18b4151980 2023-12-13T23:56:29.980 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:29.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 msgr2=0x7f18a006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 0x7f18a006d6b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f18b4050020 tx=0x7f18a4008040 comp rx=0 tx=0).stop 2023-12-13T23:56:29.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 msgr2=0x7f18b4158fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:29.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f18a8005040 tx=0x7f18a8001120 comp rx=0 tx=0).stop 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 shutdown_connections 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18a006b1f0 0x7f18a006d6b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18b415a9c0 0x7f18b415ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18b4151980 0x7f18b4158fb0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 --2- 172.21.15.124:0/79455039 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f189c0047c0 0x7f18b414d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 >> 172.21.15.124:0/79455039 conn(0x7f18b40b4720 msgr2=0x7f18b406c930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 shutdown_connections 2023-12-13T23:56:29.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:29.983+0000 7f18ba67c640 1 -- 172.21.15.124:0/79455039 wait complete. 2023-12-13T23:56:29.993 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:29.993 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:30.008 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:30.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.148+0000 7f80600f3640 1 Processor -- start 2023-12-13T23:56:30.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f80600f3640 1 -- start start 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f80600f3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f80580ceb70 0x7f80580cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f80600f3640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8058071770 con 0x7f8058144980 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f80600f3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80580ccdc0 con 0x7f80580b4370 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f80600f3640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f805815ced0 con 0x7f80580ceb70 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f805d68e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f80580ceb70 0x7f80580cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f805de8f640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8058144980 0x7f805815c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36304/0 (socket says 172.21.15.124:36304) 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.149+0000 7f805de8f640 1 -- 172.21.15.124:0/942556252 learned_addr learned my addr 172.21.15.124:0/942556252 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.150+0000 7f805d68e640 1 -- 172.21.15.124:0/942556252 >> v1:172.21.15.203:6789/0 conn(0x7f80580b4370 legacy=0x7f8058071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.150+0000 7f805d68e640 1 -- 172.21.15.124:0/942556252 >> v1:172.21.15.124:6789/0 conn(0x7f8058144980 legacy=0x7f805815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.149 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.150+0000 7f805d68e640 1 -- 172.21.15.124:0/942556252 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80580cd0a0 con 0x7f80580ceb70 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.150+0000 7f805d68e640 1 --2- 172.21.15.124:0/942556252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580ceb70 0x7f80580cc450 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f80480097a0 tx=0x7f804802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06b56a56c618516 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f805ce8d640 1 -- 172.21.15.124:0/942556252 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048031830 con 0x7f80580ceb70 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f805ce8d640 1 -- 172.21.15.124:0/942556252 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8048031990 con 0x7f80580ceb70 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f805ce8d640 1 -- 172.21.15.124:0/942556252 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048031d20 con 0x7f80580ceb70 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 -- 172.21.15.124:0/942556252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580ceb70 msgr2=0x7f80580cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 --2- 172.21.15.124:0/942556252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580ceb70 0x7f80580cc450 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f80480097a0 tx=0x7f804802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 -- 172.21.15.124:0/942556252 shutdown_connections 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 --2- 172.21.15.124:0/942556252 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580ceb70 0x7f80580cc450 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 -- 172.21.15.124:0/942556252 >> 172.21.15.124:0/942556252 conn(0x7f805806a4d0 msgr2=0x7f805806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 -- 172.21.15.124:0/942556252 shutdown_connections 2023-12-13T23:56:30.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.151+0000 7f80600f3640 1 -- 172.21.15.124:0/942556252 wait complete. 2023-12-13T23:56:30.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.152+0000 7f80600f3640 1 Processor -- start 2023-12-13T23:56:30.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.152+0000 7f80600f3640 1 -- start start 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 0x7f80580c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 0x7f80580c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058137910 con 0x7f80580b4370 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80580caef0 con 0x7f80580ceb70 2023-12-13T23:56:30.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f80600f3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8058156e40 con 0x7f8058144980 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49716/0 (socket says 172.21.15.124:49716) 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 -- 172.21.15.124:0/4082212701 learned_addr learned my addr 172.21.15.124:0/4082212701 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805d68e640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 0x7f80580c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 -- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 msgr2=0x7f80580c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805e690640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 0x7f80580c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 0x7f80580c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 -- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 msgr2=0x7f80580c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 0x7f80580c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805de8f640 1 -- 172.21.15.124:0/4082212701 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8048009400 con 0x7f80580b4370 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805d68e640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 0x7f80580c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.153+0000 7f805e690640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 0x7f80580c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f805de8f640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f805400c480 tx=0x7f805400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054005ce0 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054005e40 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80540072f0 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8058157120 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.154+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80581576e0 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.155+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80581578d0 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.155+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8054007510 con 0x7f80580b4370 2023-12-13T23:56:30.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.156+0000 7f804e7fc640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 0x7f803c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.156+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8054088fe0 con 0x7f80580b4370 2023-12-13T23:56:30.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.156+0000 7f805d68e640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 0x7f803c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.156+0000 7f805d68e640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 0x7f803c06d680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8048036000 tx=0x7f8048008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.160+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8054053cc0 con 0x7f80580b4370 2023-12-13T23:56:30.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.326+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8058069400 con 0x7f80580b4370 2023-12-13T23:56:30.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.327+0000 7f804e7fc640 1 -- 172.21.15.124:0/4082212701 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f8054017090 con 0x7f80580b4370 2023-12-13T23:56:30.327 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:30.327 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 msgr2=0x7f803c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 0x7f803c06d680 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8048036000 tx=0x7f8048008030 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 msgr2=0x7f80581373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f805400c480 tx=0x7f805400c940 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 shutdown_connections 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f803c06b1c0 0x7f803c06d680 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f80580ceb70 0x7f80580c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8058144980 0x7f80580c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 --2- 172.21.15.124:0/4082212701 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f80580b4370 0x7f80581373d0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.329+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 >> 172.21.15.124:0/4082212701 conn(0x7f805806a4d0 msgr2=0x7f8058070290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.330+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 shutdown_connections 2023-12-13T23:56:30.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.330+0000 7f80600f3640 1 -- 172.21.15.124:0/4082212701 wait complete. 2023-12-13T23:56:30.342 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:30.342 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:30.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 Processor -- start 2023-12-13T23:56:30.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 -- start start 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f90b8143490 0x7f90b80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90b80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b815e2a0 con 0x7f90b815a4e0 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b815b520 con 0x7f90b8143490 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.447+0000 7f90be353640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8143870 con 0x7f90b815e730 2023-12-13T23:56:30.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90bc8f0640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f90b815a4e0 0x7f90b815a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:36314/0 (socket says 172.21.15.124:36314) 2023-12-13T23:56:30.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90bc8f0640 1 -- 172.21.15.124:0/2221776899 learned_addr learned my addr 172.21.15.124:0/2221776899 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:30.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b77fe640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90b80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.456 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2221776899 >> v2:172.21.15.203:3300/0 conn(0x7f90b8143490 0x7f90b80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880703525 0 0) 0x7f90b815e2a0 con 0x7f90b815a4e0 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90a0003520 con 0x7f90b815a4e0 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2671512950 0 0) 0x7f90a0003520 con 0x7f90b815a4e0 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 msgr2=0x7f90b80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90b80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 >> v2:172.21.15.203:3300/0 conn(0x7f90b8143490 msgr2=0x7f90b80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 --2- 172.21.15.124:0/2221776899 >> v2:172.21.15.203:3300/0 conn(0x7f90b8143490 0x7f90b80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b815b240 con 0x7f90b815a4e0 2023-12-13T23:56:30.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f90b00019f0 con 0x7f90b815a4e0 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 >> v1:172.21.15.124:6789/0 conn(0x7f90b815a4e0 legacy=0x7f90b815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a00047c0 0x7f90a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b815b520 con 0x7f90a0003970 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b8143870 con 0x7f90a00047c0 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a000b980 con 0x7f90b815e730 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90b77fe640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a00047c0 0x7f90a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.448+0000 7f90bc8f0640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 -- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 msgr2=0x7f90a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 -- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a00047c0 msgr2=0x7f90a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90a00047c0 0x7f90a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 -- 172.21.15.124:0/2221776899 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a0003520 con 0x7f90a0003970 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90bc8f0640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b815e730 0x7f90a0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90a0003d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009ac0 tx=0x7f90a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21585a17d9a4b6cb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02c710 con 0x7f90a0003970 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac02c870 con 0x7f90a0003970 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.449+0000 7f90b6ffd640 1 -- 172.21.15.124:0/2221776899 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac02cc00 con 0x7f90a0003970 2023-12-13T23:56:30.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 -- 172.21.15.124:0/2221776899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 msgr2=0x7f90a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 --2- 172.21.15.124:0/2221776899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90a0003d90 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f90ac009ac0 tx=0x7f90a00057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90bc8f0640 1 -- 172.21.15.124:0/2221776899 reap_dead start 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 -- 172.21.15.124:0/2221776899 shutdown_connections 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 -- 172.21.15.124:0/2221776899 >> 172.21.15.124:0/2221776899 conn(0x7f90b806a4d0 msgr2=0x7f90b806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 -- 172.21.15.124:0/2221776899 shutdown_connections 2023-12-13T23:56:30.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.450+0000 7f90be353640 1 -- 172.21.15.124:0/2221776899 wait complete. 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.451+0000 7f90be353640 1 Processor -- start 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.451+0000 7f90be353640 1 -- start start 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b80c7f30 0x7f90b80ae270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b80c3fd0 0x7f90b8140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b80c9090 con 0x7f90a0003970 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90b80c8410 con 0x7f90b80c3fd0 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90be353640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b80c86f0 con 0x7f90b80c7f30 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:49734/0 (socket says 172.21.15.124:49734) 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 -- 172.21.15.124:0/2789009482 learned_addr learned my addr 172.21.15.124:0/2789009482 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 -- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b80c7f30 msgr2=0x7f90b80ae270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b80c7f30 0x7f90b80ae270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 -- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b80c3fd0 msgr2=0x7f90b8140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b80c3fd0 0x7f90b8140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.452+0000 7f90b7fff640 1 -- 172.21.15.124:0/2789009482 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b80c89d0 con 0x7f90a0003970 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90b7fff640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f90ac037040 tx=0x7f90ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac003810 con 0x7f90a0003970 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ac003970 con 0x7f90a0003970 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b80c8cb0 con 0x7f90a0003970 2023-12-13T23:56:30.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ac003d20 con 0x7f90a0003970 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.453+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b8070d20 con 0x7f90a0003970 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.454+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b8069400 con 0x7f90a0003970 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.454+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f90ac040450 con 0x7f90a0003970 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.454+0000 7f90b4ff9640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 0x7f90a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.455+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f90ac0ab880 con 0x7f90a0003970 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.455+0000 7f90b77fe640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 0x7f90a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.455+0000 7f90b77fe640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 0x7f90a406d680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90b80cb4d0 tx=0x7f90a80095d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.459+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f90ac076420 con 0x7f90a0003970 2023-12-13T23:56:30.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.625+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f90b8070d20 con 0x7f90a0003970 2023-12-13T23:56:30.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.625+0000 7f90b4ff9640 1 -- 172.21.15.124:0/2789009482 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f90b8070d20 con 0x7f90a0003970 2023-12-13T23:56:30.624 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 msgr2=0x7f90a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 0x7f90a406d680 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f90b80cb4d0 tx=0x7f90a80095d0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 msgr2=0x7f90b8070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f90ac037040 tx=0x7f90ac004730 comp rx=0 tx=0).stop 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 shutdown_connections 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f90a406b1c0 0x7f90a406d680 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f90b80c3fd0 0x7f90b8140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f90b80c7f30 0x7f90b80ae270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 --2- 172.21.15.124:0/2789009482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90a0003970 0x7f90b8070600 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 >> 172.21.15.124:0/2789009482 conn(0x7f90b806a4d0 msgr2=0x7f90b80ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 shutdown_connections 2023-12-13T23:56:30.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.627+0000 7f90be353640 1 -- 172.21.15.124:0/2789009482 wait complete. 2023-12-13T23:56:30.637 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:30.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.740+0000 7fea1cbb6640 1 Processor -- start 2023-12-13T23:56:30.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.740+0000 7fea1cbb6640 1 -- start start 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1cbb6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea1815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1cbb6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fea1815e730 0x7fea1815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1cbb6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea1815e2a0 con 0x7fea18143490 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1cbb6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea1815b520 con 0x7fea1815e730 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1cbb6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea18143870 con 0x7fea180b1400 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1659c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fea18143490 0x7fea180b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:33690/0 (socket says 172.21.15.124:33690) 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1659c640 1 -- 172.21.15.124:0/2641109267 learned_addr learned my addr 172.21.15.124:0/2641109267 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.741+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226694492 0 0) 0x7fea1815e2a0 con 0x7fea18143490 2023-12-13T23:56:30.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea00003520 con 0x7fea18143490 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea16d9d640 1 --2- 172.21.15.124:0/2641109267 >> v2:172.21.15.203:3300/0 conn(0x7fea1815e730 0x7fea1815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea15d9b640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea1815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2973509572 0 0) 0x7fea00003520 con 0x7fea18143490 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 msgr2=0x7fea1815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea1815a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 >> v2:172.21.15.203:3300/0 conn(0x7fea1815e730 msgr2=0x7fea1815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 --2- 172.21.15.124:0/2641109267 >> v2:172.21.15.203:3300/0 conn(0x7fea1815e730 0x7fea1815aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea1815b240 con 0x7fea18143490 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fea080019f0 con 0x7fea18143490 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 >> v1:172.21.15.124:6789/0 conn(0x7fea18143490 legacy=0x7fea180b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea00003970 0x7fea00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00004d00 0x7fea00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1815b520 con 0x7fea00003970 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea18143870 con 0x7fea00004d00 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea0000ba30 con 0x7fea180b1400 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea00003970 0x7fea00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea15d9b640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea16d9d640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00004d00 0x7fea00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 -- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 msgr2=0x7fea000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180b1400 0x7fea000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 -- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00004d00 msgr2=0x7fea00005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00004d00 0x7fea00005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.742+0000 7fea1659c640 1 -- 172.21.15.124:0/2641109267 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea00003520 con 0x7fea00003970 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.743+0000 7fea1659c640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea00003970 0x7fea00003d90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fea0800a890 tx=0x7fea000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbaa2b02d335c58e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.743+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08034630 con 0x7fea00003970 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.743+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0802f070 con 0x7fea00003970 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.743+0000 7fea1559a640 1 -- 172.21.15.124:0/2641109267 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08034a10 con 0x7fea00003970 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.743+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2641109267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea00003970 msgr2=0x7fea00003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2641109267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea00003970 0x7fea00003d90 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fea0800a890 tx=0x7fea000057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea16d9d640 1 -- 172.21.15.124:0/2641109267 reap_dead start 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2641109267 shutdown_connections 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2641109267 >> 172.21.15.124:0/2641109267 conn(0x7fea1806a4d0 msgr2=0x7fea1806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2641109267 shutdown_connections 2023-12-13T23:56:30.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.744+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2641109267 wait complete. 2023-12-13T23:56:30.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.745+0000 7fea1cbb6640 1 Processor -- start 2023-12-13T23:56:30.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.745+0000 7fea1cbb6640 1 -- start start 2023-12-13T23:56:30.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00003970 0x7fea180b0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180af470 0x7fea180b3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea180b0980 con 0x7fea1813bde0 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea180afd70 con 0x7fea00003970 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1cbb6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea180b0050 con 0x7fea180af470 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38924/0 (socket says 172.21.15.124:38924) 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 -- 172.21.15.124:0/2384117245 learned_addr learned my addr 172.21.15.124:0/2384117245 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea1659c640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00003970 0x7fea180b0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea16d9d640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180af470 0x7fea180b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 -- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180af470 msgr2=0x7fea180b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180af470 0x7fea180b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 -- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00003970 msgr2=0x7fea180b0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00003970 0x7fea180b0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 -- 172.21.15.124:0/2384117245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea180b0330 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea15d9b640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00b7f0 tx=0x7fea0c00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c010830 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.746+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c010990 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.747+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea180b0610 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.747+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea0c010d20 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.747+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea180afb00 con 0x7fea1813bde0 2023-12-13T23:56:30.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.747+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea180b07a0 con 0x7fea1813bde0 2023-12-13T23:56:30.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.748+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fea0c014430 con 0x7fea1813bde0 2023-12-13T23:56:30.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.748+0000 7fea06ffd640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 0x7fe9f806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:30.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.748+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fea0c08a000 con 0x7fea1813bde0 2023-12-13T23:56:30.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.748+0000 7fea1659c640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 0x7fe9f806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:30.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.749+0000 7fea1659c640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 0x7fe9f806d6b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea08034000 tx=0x7fea08009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:30.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.752+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fea0c054ce0 con 0x7fea1813bde0 2023-12-13T23:56:30.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.916+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fea18069400 con 0x7fea1813bde0 2023-12-13T23:56:30.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.917+0000 7fea06ffd640 1 -- 172.21.15.124:0/2384117245 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7fea0c058300 con 0x7fea1813bde0 2023-12-13T23:56:30.916 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 msgr2=0x7fe9f806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 0x7fe9f806d6b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fea08034000 tx=0x7fea08009030 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 msgr2=0x7fea18159a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fea0c00b7f0 tx=0x7fea0c00bcb0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 shutdown_connections 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe9f806b1f0 0x7fe9f806d6b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fea180af470 0x7fea180b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea1813bde0 0x7fea18159a60 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 --2- 172.21.15.124:0/2384117245 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fea00003970 0x7fea180b0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 >> 172.21.15.124:0/2384117245 conn(0x7fea1806a4d0 msgr2=0x7fea180b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:30.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 shutdown_connections 2023-12-13T23:56:30.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:30.919+0000 7fea1cbb6640 1 -- 172.21.15.124:0/2384117245 wait complete. 2023-12-13T23:56:30.928 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:31.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.033+0000 7f8cddc31640 1 Processor -- start 2023-12-13T23:56:31.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.033+0000 7f8cddc31640 1 -- start start 2023-12-13T23:56:31.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd815d330 0x7f8cd80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8cd80d2910 0x7f8cd80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 0x7f8cd80d3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd80b2b10 con 0x7f8cd80d2910 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd80d1590 con 0x7f8cd80d2d90 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cddc31640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80d1870 con 0x7f8cd815d330 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd6ffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8cd80d2910 0x7f8cd80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd6ffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8cd80d2910 0x7f8cd80d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38932/0 (socket says 172.21.15.124:38932) 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd6ffd640 1 -- 172.21.15.124:0/2946785927 learned_addr learned my addr 172.21.15.124:0/2946785927 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd77fe640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd815d330 0x7f8cd80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd6ffd640 1 -- 172.21.15.124:0/2946785927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd815d330 msgr2=0x7f8cd80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd7fff640 1 --2- 172.21.15.124:0/2946785927 >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 0x7f8cd80d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.034+0000 7f8cd6ffd640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd815d330 0x7f8cd80b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd6ffd640 1 -- 172.21.15.124:0/2946785927 >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 msgr2=0x7f8cd80d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:31.034 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd6ffd640 1 --2- 172.21.15.124:0/2946785927 >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 0x7f8cd80d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.035 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd6ffd640 1 -- 172.21.15.124:0/2946785927 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd80d1b50 con 0x7f8cd80d2910 2023-12-13T23:56:31.035 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd7fff640 1 --2- 172.21.15.124:0/2946785927 >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 0x7f8cd80d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd6ffd640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd80d31f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80d1540 tx=0x7f8ccc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c26d6c74b2244f19 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd67fc640 1 -- 172.21.15.124:0/2946785927 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc0317e0 con 0x7f8cd80d2910 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd67fc640 1 -- 172.21.15.124:0/2946785927 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc031940 con 0x7f8cd80d2910 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.035+0000 7f8cd67fc640 1 -- 172.21.15.124:0/2946785927 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc031cd0 con 0x7f8cd80d2910 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 -- 172.21.15.124:0/2946785927 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 msgr2=0x7f8cd80d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.036 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd80d31f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f8cd80d1540 tx=0x7f8ccc02b880 comp rx=0 tx=0).stop 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 -- 172.21.15.124:0/2946785927 shutdown_connections 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd80d31f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 --2- 172.21.15.124:0/2946785927 >> v2:172.21.15.203:3300/0 conn(0x7f8cd80d2d90 0x7f8cd80d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 --2- 172.21.15.124:0/2946785927 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd815d330 0x7f8cd80b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 -- 172.21.15.124:0/2946785927 >> 172.21.15.124:0/2946785927 conn(0x7f8cd8072020 msgr2=0x7f8cd8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 -- 172.21.15.124:0/2946785927 shutdown_connections 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.036+0000 7f8cddc31640 1 -- 172.21.15.124:0/2946785927 wait complete. 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.037+0000 7f8cddc31640 1 Processor -- start 2023-12-13T23:56:31.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.037+0000 7f8cddc31640 1 -- start start 2023-12-13T23:56:31.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd80d2d90 0x7f8cd815ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8cd815d330 0x7f8cd815e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815e5a0 con 0x7f8cd80d2910 2023-12-13T23:56:31.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd8154ee0 con 0x7f8cd815d330 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cddc31640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd815d920 con 0x7f8cd80d2d90 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38946/0 (socket says 172.21.15.124:38946) 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 -- 172.21.15.124:0/1167820298 learned_addr learned my addr 172.21.15.124:0/1167820298 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 -- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd80d2d90 msgr2=0x7f8cd815ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd80d2d90 0x7f8cd815ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd7fff640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8cd815d330 0x7f8cd815e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 -- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8cd815d330 msgr2=0x7f8cd815e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8cd815d330 0x7f8cd815e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.038+0000 7f8cd77fe640 1 -- 172.21.15.124:0/1167820298 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc0094d0 con 0x7f8cd80d2910 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cd77fe640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c7f0 tx=0x7f8cc800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8004220 con 0x7f8cd80d2910 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc80043a0 con 0x7f8cd80d2910 2023-12-13T23:56:31.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd815dc00 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8007860 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.039+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd815e1c0 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.040+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd8069400 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.040+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8cc8007bd0 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.040+0000 7f8cbffff640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 0x7f8cc406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.041+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8cc8089ae0 con 0x7f8cd80d2910 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.041+0000 7f8cd6ffd640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 0x7f8cc406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.041+0000 7f8cd6ffd640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 0x7f8cc406d680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc036000 tx=0x7f8ccc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.045+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8cc8054710 con 0x7f8cd80d2910 2023-12-13T23:56:31.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.212+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8cd8072780 con 0x7f8cd80d2910 2023-12-13T23:56:31.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.212+0000 7f8cbffff640 1 -- 172.21.15.124:0/1167820298 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8cc801e020 con 0x7f8cd80d2910 2023-12-13T23:56:31.211 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.214+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 msgr2=0x7f8cc406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.214+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 0x7f8cc406d680 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc036000 tx=0x7f8ccc030000 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.214+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 msgr2=0x7f8cd81549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.214+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8cc800c7f0 tx=0x7f8cc800ccb0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 shutdown_connections 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8cc406b1c0 0x7f8cc406d680 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8cd815d330 0x7f8cd815e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8cd80d2d90 0x7f8cd815ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 --2- 172.21.15.124:0/1167820298 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8cd80d2910 0x7f8cd81549a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 >> 172.21.15.124:0/1167820298 conn(0x7f8cd8072020 msgr2=0x7f8cd80b6720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 shutdown_connections 2023-12-13T23:56:31.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.215+0000 7f8cddc31640 1 -- 172.21.15.124:0/1167820298 wait complete. 2023-12-13T23:56:31.224 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:31.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.327+0000 7fa32423d640 1 Processor -- start 2023-12-13T23:56:31.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 -- start start 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c15d330 0x7fa31c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa31c0d2910 0x7fa31c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31c0b2b10 con 0x7fa31c0d2d90 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa31c0d1590 con 0x7fa31c0d2910 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa32423d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c0d1870 con 0x7fa31c15d330 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa3217d8640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa31c0d2910 0x7fa31c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.328+0000 7fa3227da640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa31c0d2d90 0x7fa31c0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:33704/0 (socket says 172.21.15.124:33704) 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa3227da640 1 -- 172.21.15.124:0/1557649639 learned_addr learned my addr 172.21.15.124:0/1557649639 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa321fd9640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c15d330 0x7fa31c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa3217d8640 1 -- 172.21.15.124:0/1557649639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c15d330 msgr2=0x7fa31c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:31.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa3217d8640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c15d330 0x7fa31c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa3217d8640 1 -- 172.21.15.124:0/1557649639 >> v1:172.21.15.124:6789/0 conn(0x7fa31c0d2d90 legacy=0x7fa31c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa320fd7640 1 -- 172.21.15.124:0/1557649639 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906552328 0 0) 0x7fa31c0b2b10 con 0x7fa31c0d2d90 2023-12-13T23:56:31.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.329+0000 7fa3217d8640 1 -- 172.21.15.124:0/1557649639 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31c0d1b50 con 0x7fa31c0d2910 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa3217d8640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c0d31f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0097a0 tx=0x7fa30c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef921a4503b3810 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa320fd7640 1 -- 172.21.15.124:0/1557649639 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c02e070 con 0x7fa31c0d2910 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa320fd7640 1 -- 172.21.15.124:0/1557649639 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30c0315a0 con 0x7fa31c0d2910 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa320fd7640 1 -- 172.21.15.124:0/1557649639 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30c031930 con 0x7fa31c0d2910 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 -- 172.21.15.124:0/1557649639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 msgr2=0x7fa31c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c0d31f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa30c0097a0 tx=0x7fa30c02b840 comp rx=0 tx=0).stop 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 -- 172.21.15.124:0/1557649639 shutdown_connections 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c0d31f0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 --2- 172.21.15.124:0/1557649639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c15d330 0x7fa31c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 -- 172.21.15.124:0/1557649639 >> 172.21.15.124:0/1557649639 conn(0x7fa31c072020 msgr2=0x7fa31c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 -- 172.21.15.124:0/1557649639 shutdown_connections 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.330+0000 7fa32423d640 1 -- 172.21.15.124:0/1557649639 wait complete. 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.331+0000 7fa32423d640 1 Processor -- start 2023-12-13T23:56:31.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 -- start start 2023-12-13T23:56:31.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c0d2d90 0x7fa31c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa31c15d330 0x7fa31c15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c15f1a0 con 0x7fa31c0d2910 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c1f3240 con 0x7fa31c15d330 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa32423d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31c1f3520 con 0x7fa31c0d2d90 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa321fd9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa321fd9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38958/0 (socket says 172.21.15.124:38958) 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.332+0000 7fa321fd9640 1 -- 172.21.15.124:0/3790518426 learned_addr learned my addr 172.21.15.124:0/3790518426 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa3227da640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa31c15d330 0x7fa31c15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa3217d8640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c0d2d90 0x7fa31c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 -- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c0d2d90 msgr2=0x7fa31c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c0d2d90 0x7fa31c14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 -- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa31c15d330 msgr2=0x7fa31c15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa31c15d330 0x7fa31c15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 -- 172.21.15.124:0/3790518426 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30c009400 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa321fd9640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa318007ef0 tx=0x7fa31800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa318014620 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa318014780 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31c1f3800 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa318014b30 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.333+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31c1f2f30 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.334+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa31c0b4ad0 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.334+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa318014c90 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.335+0000 7fa3127fc640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 0x7fa30006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.335+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fa31808a540 con 0x7fa31c0d2910 2023-12-13T23:56:31.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.335+0000 7fa3217d8640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 0x7fa30006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.335+0000 7fa3217d8640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 0x7fa30006d680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa30c036000 tx=0x7fa30c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.339+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa318055170 con 0x7fa31c0d2910 2023-12-13T23:56:31.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.506+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7fa31c15f4e0 con 0x7fa31c0d2910 2023-12-13T23:56:31.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.507+0000 7fa3127fc640 1 -- 172.21.15.124:0/3790518426 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7fa318058790 con 0x7fa31c0d2910 2023-12-13T23:56:31.506 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 msgr2=0x7fa30006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 0x7fa30006d680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa30c036000 tx=0x7fa30c008030 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 msgr2=0x7fa31c15e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa318007ef0 tx=0x7fa31800e4b0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 shutdown_connections 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa30006b1c0 0x7fa30006d680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa31c15d330 0x7fa31c15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa31c0d2d90 0x7fa31c14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 --2- 172.21.15.124:0/3790518426 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa31c0d2910 0x7fa31c15e450 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 >> 172.21.15.124:0/3790518426 conn(0x7fa31c072020 msgr2=0x7fa31c0b7080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 shutdown_connections 2023-12-13T23:56:31.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.509+0000 7fa32423d640 1 -- 172.21.15.124:0/3790518426 wait complete. 2023-12-13T23:56:31.518 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:31.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.620+0000 7f11cc789640 1 Processor -- start 2023-12-13T23:56:31.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.620+0000 7f11cc789640 1 -- start start 2023-12-13T23:56:31.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11c4146b60 0x7f11c4161870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 0x7f11c40cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c406bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40c5e80 con 0x7f11c4146b60 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c4146f40 con 0x7f11c4070980 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cc789640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40c6650 con 0x7f11c40cbb70 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11c4146b60 0x7f11c4161870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f11c4146b60 0x7f11c4161870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38960/0 (socket says 172.21.15.124:38960) 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 -- 172.21.15.124:0/771272248 learned_addr learned my addr 172.21.15.124:0/771272248 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11c9d24640 1 --2- 172.21.15.124:0/771272248 >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 0x7f11c40cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 -- 172.21.15.124:0/771272248 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 msgr2=0x7f11c406bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cad26640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c406bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c406bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 -- 172.21.15.124:0/771272248 >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 msgr2=0x7f11c40cb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 --2- 172.21.15.124:0/771272248 >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 0x7f11c40cb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11ca525640 1 -- 172.21.15.124:0/771272248 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c4161db0 con 0x7f11c4146b60 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11c9d24640 1 --2- 172.21.15.124:0/771272248 >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 0x7f11c40cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.621+0000 7f11cad26640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c406bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11ca525640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c4146b60 0x7f11c4161870 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f11c4067e90 tx=0x7f11bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d34e5044289155 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11c9523640 1 -- 172.21.15.124:0/771272248 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc02ca70 con 0x7f11c4146b60 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11c9523640 1 -- 172.21.15.124:0/771272248 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc02cbd0 con 0x7f11c4146b60 2023-12-13T23:56:31.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11c9523640 1 -- 172.21.15.124:0/771272248 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc0343f0 con 0x7f11c4146b60 2023-12-13T23:56:31.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 -- 172.21.15.124:0/771272248 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c4146b60 msgr2=0x7f11c4161870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c4146b60 0x7f11c4161870 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f11c4067e90 tx=0x7f11bc02bed0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 -- 172.21.15.124:0/771272248 shutdown_connections 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c406bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 --2- 172.21.15.124:0/771272248 >> v2:172.21.15.203:3300/0 conn(0x7f11c4070980 0x7f11c40cb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 --2- 172.21.15.124:0/771272248 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c4146b60 0x7f11c4161870 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.622+0000 7f11cc789640 1 -- 172.21.15.124:0/771272248 >> 172.21.15.124:0/771272248 conn(0x7f11c40b2c10 msgr2=0x7f11c40b3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.623+0000 7f11cc789640 1 -- 172.21.15.124:0/771272248 shutdown_connections 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.623+0000 7f11cc789640 1 -- 172.21.15.124:0/771272248 wait complete. 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 Processor -- start 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 -- start start 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c4137ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 0x7f11c40b5340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c415f100 con 0x7f11c41347e0 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40b5880 con 0x7f11c4146b60 2023-12-13T23:56:31.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.624+0000 7f11cc789640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c415f3a0 con 0x7f11c40cbb70 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38974/0 (socket says 172.21.15.124:38974) 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 -- 172.21.15.124:0/1871860700 learned_addr learned my addr 172.21.15.124:0/1871860700 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11c9d24640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 0x7f11c40b5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11ca525640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c4137ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 -- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 msgr2=0x7f11c4137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c4137ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 -- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 msgr2=0x7f11c40b5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 0x7f11c40b5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 -- 172.21.15.124:0/1871860700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009b30 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11c9d24640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 0x7f11c40b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cad26640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c7f0 tx=0x7f11c000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0007820 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c0007980 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c415f680 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11c0007d70 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.625+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c415fc40 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.626+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c40b3790 con 0x7f11c41347e0 2023-12-13T23:56:31.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.627+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f11c00113f0 con 0x7f11c41347e0 2023-12-13T23:56:31.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.627+0000 7f11baffd640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 0x7f11a806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:31.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.627+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f11c0089ee0 con 0x7f11c41347e0 2023-12-13T23:56:31.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.627+0000 7f11ca525640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 0x7f11a806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:31.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.628+0000 7f11ca525640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 0x7f11a806d6b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11bc035000 tx=0x7f11bc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:31.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.632+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f11c0054a80 con 0x7f11c41347e0 2023-12-13T23:56:31.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.808+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f11c40cfd30 con 0x7f11c41347e0 2023-12-13T23:56:31.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.808+0000 7f11baffd640 1 -- 172.21.15.124:0/1871860700 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7f11c001f080 con 0x7f11c41347e0 2023-12-13T23:56:31.807 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:31.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 msgr2=0x7f11a806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 0x7f11a806d6b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f11bc035000 tx=0x7f11bc030000 comp rx=0 tx=0).stop 2023-12-13T23:56:31.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 msgr2=0x7f11c4131b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:31.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f11c000c7f0 tx=0x7f11c000ccb0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 shutdown_connections 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f11a806b1f0 0x7f11a806d6b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f11c4146b60 0x7f11c40b5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f11c41347e0 0x7f11c4131b00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 --2- 172.21.15.124:0/1871860700 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f11c40cbb70 0x7f11c4137ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 >> 172.21.15.124:0/1871860700 conn(0x7f11c40b2c10 msgr2=0x7f11c406c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 shutdown_connections 2023-12-13T23:56:31.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:31.810+0000 7f11cc789640 1 -- 172.21.15.124:0/1871860700 wait complete. 2023-12-13T23:56:31.819 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:56:31.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:31.820 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:31.845 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[103467]: fuse finished with error 0 and tester_r 0 2023-12-13T23:56:31.846 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:37.948 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:56:37.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:37.949 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:56:37.967 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:56:38.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.115+0000 7f37c8dc5640 1 Processor -- start 2023-12-13T23:56:38.114 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.115+0000 7f37c8dc5640 1 -- start start 2023-12-13T23:56:38.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c8dc5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:38.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c8dc5640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c4071770 con 0x7f37c40ceb70 2023-12-13T23:56:38.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c8dc5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37c40ccb60 con 0x7f37c40b4370 2023-12-13T23:56:38.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c8dc5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40ccdc0 con 0x7f37c4144980 2023-12-13T23:56:38.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c1d9b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f37c40ceb70 0x7f37c40cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:33714/0 (socket says 172.21.15.124:33714) 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c1d9b640 1 -- 172.21.15.124:0/2953255141 learned_addr learned my addr 172.21.15.124:0/2953255141 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c259c640 1 --2- 172.21.15.124:0/2953255141 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1276126493 0 0) 0x7f37c4071770 con 0x7f37c40ceb70 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ac003520 con 0x7f37c40ceb70 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2655699372 0 0) 0x7f37c40ccb60 con 0x7f37c40b4370 2023-12-13T23:56:38.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c4071770 con 0x7f37c40b4370 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c259c640 1 -- 172.21.15.124:0/2953255141 >> v1:172.21.15.203:6789/0 conn(0x7f37c40b4370 legacy=0x7f37c4071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.116+0000 7f37c259c640 1 -- 172.21.15.124:0/2953255141 >> v1:172.21.15.124:6789/0 conn(0x7f37c40ceb70 legacy=0x7f37c40cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.117+0000 7f37c259c640 1 -- 172.21.15.124:0/2953255141 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c40cd0a0 con 0x7f37c4144980 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.117+0000 7f37c259c640 1 --2- 172.21.15.124:0/2953255141 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c415c7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ecbc2b2e3880ca43 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.117+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc02ccd0 con 0x7f37c4144980 2023-12-13T23:56:38.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc02ce30 con 0x7f37c4144980 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c159a640 1 -- 172.21.15.124:0/2953255141 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc034640 con 0x7f37c4144980 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 -- 172.21.15.124:0/2953255141 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 msgr2=0x7f37c415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/2953255141 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c415c7c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f37bc009dc0 tx=0x7f37bc02bf20 comp rx=0 tx=0).stop 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 -- 172.21.15.124:0/2953255141 shutdown_connections 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/2953255141 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c415c7c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 -- 172.21.15.124:0/2953255141 >> 172.21.15.124:0/2953255141 conn(0x7f37c406a4d0 msgr2=0x7f37c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 -- 172.21.15.124:0/2953255141 shutdown_connections 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.118+0000 7f37c8dc5640 1 -- 172.21.15.124:0/2953255141 wait complete. 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.119+0000 7f37c8dc5640 1 Processor -- start 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.119+0000 7f37c8dc5640 1 -- start start 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37c40ceb70 0x7f37c40ae390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:38.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c40b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c41370e0 con 0x7f37c40b4370 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40b0b90 con 0x7f37c40ceb70 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c8dc5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37c40aff80 con 0x7f37c4144980 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c259c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:38984/0 (socket says 172.21.15.124:38984) 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.120+0000 7f37c259c640 1 -- 172.21.15.124:0/3330036893 learned_addr learned my addr 172.21.15.124:0/3330036893 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:38.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c1d9b640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37c40ceb70 0x7f37c40ae390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c2d9d640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c40b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 -- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 msgr2=0x7f37c40b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c40b1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 -- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37c40ceb70 msgr2=0x7f37c40ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37c40ceb70 0x7f37c40ae390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 -- 172.21.15.124:0/3330036893 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37bc009a70 con 0x7f37c40b4370 2023-12-13T23:56:38.130 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c259c640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f37bc02dfd0 tx=0x7f37bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:38.130 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc007000 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37bc03e630 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c40b0260 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37bc03e9e0 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.121+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c40b0820 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.122+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c40b09b0 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.122+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f37bc03b050 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.123+0000 7f37b2ffd640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 0x7f37a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.123+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f37bc0ac4b0 con 0x7f37c40b4370 2023-12-13T23:56:38.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.123+0000 7f37c1d9b640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 0x7f37a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:38.134 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.123+0000 7f37c1d9b640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 0x7f37a406d680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f37b8000f60 tx=0x7f37b8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:38.134 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.127+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f37bc077200 con 0x7f37c40b4370 2023-12-13T23:56:38.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.326+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f37c4069400 con 0x7f37c40b4370 2023-12-13T23:56:38.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.326+0000 7f37b2ffd640 1 -- 172.21.15.124:0/3330036893 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+82 (secure 0 0 0) 0x7f37bc009450 con 0x7f37c40b4370 2023-12-13T23:56:38.325 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 msgr2=0x7f37a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 0x7f37a406d680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f37b8000f60 tx=0x7f37b8009040 comp rx=0 tx=0).stop 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 msgr2=0x7f37c40ca700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f37bc02dfd0 tx=0x7f37bc004a40 comp rx=0 tx=0).stop 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 shutdown_connections 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f37a406b1c0 0x7f37a406d680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f37c40ceb70 0x7f37c40ae390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f37c4144980 0x7f37c40b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 --2- 172.21.15.124:0/3330036893 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37c40b4370 0x7f37c40ca700 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:38.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 >> 172.21.15.124:0/3330036893 conn(0x7f37c406a4d0 msgr2=0x7f37c4158510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:38.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 shutdown_connections 2023-12-13T23:56:38.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:38.328+0000 7f37c8dc5640 1 -- 172.21.15.124:0/3330036893 wait complete. 2023-12-13T23:56:38.338 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:56:38.338 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:56:38.338 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:56:38.338 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:56:38.339 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:38.339 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:56:38.339 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:56:38.339 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:56:38.339 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:38.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.340 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.356 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42608sec preferred_lft 42608sec 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:56:38.357 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.358 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:56:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:38.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.359 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:56:38.422 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:56:38.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.424 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:56:38.479 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:56:38.480 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:56:38.480 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:56:38.480 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:56:38.480 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:56:38.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.480 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:56:38.480 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:56:38.480 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:56:38.480 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:56:38.480 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:56:38.481 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:56:38.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.575 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:56:38.591 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:56:38.591 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:56:38.591 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:38.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.591 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:56:38.657 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:56:38.657 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:38.658 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:38.712 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:38.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:38.713 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:38.767 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:38.834 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:38.890 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:38.890 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:38.890 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:38.891 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:38.892 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:38.893 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:38.893 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:38.893 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:38.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:38.893 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:38.945 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:56:38.946 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:56:38.988 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:39.016 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:39.072 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:39.073 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:39.074 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:39.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.075 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:39.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:56:39.075+0000 7f6172c4a480 -1 init, newargv = 0x55a2cdb69030 newargc=15 2023-12-13T23:56:39.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[104701]: starting ceph client 2023-12-13T23:56:39.078 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[104701]: starting fuse 2023-12-13T23:56:39.090 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:56:39.091 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:56:39.091 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:56:39.092 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:56:39.093 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:56:39.093 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:39.093 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:56:39.093 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:56:39.210 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.104701.asok 2023-12-13T23:56:39.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:56:39.221 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.104701.asok 2023-12-13T23:56:39.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.221 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.104701.asok status 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "104701", 2023-12-13T23:56:39.358 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5565, 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5565 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1295308179 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:39.359 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1295308179 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5565 192.168.0.1:0/1295308179", 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/1295308179", 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 15, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 38, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:56:39.360 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:56:39.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.369 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:56:39.369 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:39.385 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:56:39.386 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:39.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.386 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:39.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.453 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2023-12-13T23:56:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:56:39.527 INFO:teuthology.orchestra.run.smithi124.stderr:setfattr: layoutfile: Permission denied 2023-12-13T23:56:39.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:39.527 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2023-12-13T23:56:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:56:39.582 INFO:teuthology.orchestra.run.smithi124.stderr:setfattr: layoutdir: Permission denied 2023-12-13T23:56:39.583 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:39.583 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:39.638 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.780+0000 7f0c05f4c640 1 Processor -- start 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.780+0000 7f0c05f4c640 1 -- start start 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0c05f4c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0c0015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0c05f4c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00071770 con 0x7f0c000ceb70 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0c05f4c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c000ccb60 con 0x7f0c000b4370 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0c05f4c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000ccdc0 con 0x7f0c00144980 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bfeffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0c000ceb70 0x7f0c000cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:33726/0 (socket says 172.21.15.124:33726) 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bfeffd640 1 -- 172.21.15.124:0/1391863480 learned_addr learned my addr 172.21.15.124:0/1391863480 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:39.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239126233 0 0) 0x7f0c00071770 con 0x7f0c000ceb70 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0be8003520 con 0x7f0c000ceb70 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0c0015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.781+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1730178616 0 0) 0x7f0be8003520 con 0x7f0c000ceb70 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 msgr2=0x7f0c0015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0c0015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 >> v1:172.21.15.203:6789/0 conn(0x7f0c000b4370 legacy=0x7f0c00071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000cd0a0 con 0x7f0c000ceb70 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0bf4001670 con 0x7f0c000ceb70 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0c0015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 >> v1:172.21.15.124:6789/0 conn(0x7f0c000ceb70 legacy=0x7f0c000cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80039a0 0x7f0be8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0be80047c0 0x7f0be8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0be8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000ccdc0 con 0x7f0be80047c0 2023-12-13T23:56:39.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be800ba80 con 0x7f0be80039a0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be800bc40 con 0x7f0c00144980 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0be80047c0 0x7f0be8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80039a0 0x7f0be8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfffff640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0be8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 -- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 msgr2=0x7f0be8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0be8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 -- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80039a0 msgr2=0x7f0be8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80039a0 0x7f0be8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfeffd640 1 -- 172.21.15.124:0/1391863480 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be8003520 con 0x7f0be80047c0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.782+0000 7f0bfffff640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c00144980 0x7f0be8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0be80047c0 0x7f0be8004c40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c210 tx=0x7f0be80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f63a2d70f21baa5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4005ad0 con 0x7f0be80047c0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf4005c30 con 0x7f0be80047c0 2023-12-13T23:56:39.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0bfe7fc640 1 -- 172.21.15.124:0/1391863480 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf402c890 con 0x7f0be80047c0 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0c05f4c640 1 -- 172.21.15.124:0/1391863480 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0be80047c0 msgr2=0x7f0be8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/1391863480 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0be80047c0 0x7f0be8004c40 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0bf400c210 tx=0x7f0be80057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0bfffff640 1 -- 172.21.15.124:0/1391863480 reap_dead start 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0c05f4c640 1 -- 172.21.15.124:0/1391863480 shutdown_connections 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.783+0000 7f0c05f4c640 1 -- 172.21.15.124:0/1391863480 >> 172.21.15.124:0/1391863480 conn(0x7f0c0006a4d0 msgr2=0x7f0c0006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.784+0000 7f0c05f4c640 1 -- 172.21.15.124:0/1391863480 shutdown_connections 2023-12-13T23:56:39.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.784+0000 7f0c05f4c640 1 -- 172.21.15.124:0/1391863480 wait complete. 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 Processor -- start 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 -- start start 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80047c0 0x7f0c001404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 0x7f0c0015a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.785+0000 7f0c05f4c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000cb760 con 0x7f0c000c9d00 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0c05f4c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0015aad0 con 0x7f0be80047c0 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0c05f4c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000cab50 con 0x7f0c0015a130 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39008/0 (socket says 172.21.15.124:39008) 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 -- 172.21.15.124:0/3213035687 learned_addr learned my addr 172.21.15.124:0/3213035687 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80047c0 0x7f0c001404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 0x7f0c0015a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 -- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 msgr2=0x7f0c0015a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 0x7f0c0015a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 -- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80047c0 msgr2=0x7f0c001404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80047c0 0x7f0c001404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 -- 172.21.15.124:0/3213035687 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000cae30 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfeffd640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 0x7f0c0015a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0bfffff640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0be800bdd0 tx=0x7f0bf800eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8017070 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf8006b50 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000cb110 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf8006d00 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.786+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c00140a10 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.787+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c00140bd0 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.788+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0bf8003970 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.788+0000 7f0be7fff640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 0x7f0bec06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.788+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f0bf8089290 con 0x7f0c000c9d00 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.788+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 0x7f0bec06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:39.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.789+0000 7f0bff7fe640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 0x7f0bec06d680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0006fa0 tx=0x7f0bf0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:39.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.792+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0bf8053db0 con 0x7f0c000c9d00 2023-12-13T23:56:39.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.962+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c00155b10 con 0x7f0c000c9d00 2023-12-13T23:56:39.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.964+0000 7f0be7fff640 1 -- 172.21.15.124:0/3213035687 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f0bf80573d0 con 0x7f0c000c9d00 2023-12-13T23:56:39.963 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:39.963 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.965+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 msgr2=0x7f0bec06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.965+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 0x7f0bec06d680 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0006fa0 tx=0x7f0bf0008040 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.965+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 msgr2=0x7f0c000ca180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.965+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f0be800bdd0 tx=0x7f0bf800eb90 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 shutdown_connections 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0bec06b1c0 0x7f0bec06d680 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c000c9d00 0x7f0c000ca180 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0c0015a130 0x7f0c0015a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 --2- 172.21.15.124:0/3213035687 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0be80047c0 0x7f0c001404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 >> 172.21.15.124:0/3213035687 conn(0x7f0c0006a4d0 msgr2=0x7f0c00157110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 shutdown_connections 2023-12-13T23:56:39.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:39.966+0000 7f0c05f4c640 1 -- 172.21.15.124:0/3213035687 wait complete. 2023-12-13T23:56:39.976 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:39.976 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:40.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.080+0000 7f54caae5640 1 Processor -- start 2023-12-13T23:56:40.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.080+0000 7f54caae5640 1 -- start start 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54caae5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f54c4150560 0x7f54c40b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54caae5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54c40b3390 0x7f54c415d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54caae5640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c40b49f0 con 0x7f54c4150940 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54caae5640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c40b3de0 con 0x7f54c40b3390 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54caae5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c40b40c0 con 0x7f54c4150560 2023-12-13T23:56:40.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54c40b3390 0x7f54c415d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c9082640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f54c4150940 0x7f54c40ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:33742/0 (socket says 172.21.15.124:33742) 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c3fff640 1 --2- 172.21.15.124:0/1018273991 >> v2:172.21.15.124:3300/0 conn(0x7f54c40b3390 0x7f54c415d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39018/0 (socket says 172.21.15.124:39018) 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c9082640 1 -- 172.21.15.124:0/1018273991 learned_addr learned my addr 172.21.15.124:0/1018273991 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c8881640 1 --2- 172.21.15.124:0/1018273991 >> v2:172.21.15.203:3300/0 conn(0x7f54c4150560 0x7f54c40b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c3fff640 1 -- 172.21.15.124:0/1018273991 >> v2:172.21.15.203:3300/0 conn(0x7f54c4150560 msgr2=0x7f54c40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c3fff640 1 --2- 172.21.15.124:0/1018273991 >> v2:172.21.15.203:3300/0 conn(0x7f54c4150560 0x7f54c40b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.081+0000 7f54c3fff640 1 -- 172.21.15.124:0/1018273991 >> v1:172.21.15.124:6789/0 conn(0x7f54c4150940 legacy=0x7f54c40ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c3fff640 1 -- 172.21.15.124:0/1018273991 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c40b43a0 con 0x7f54c40b3390 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c37fe640 1 -- 172.21.15.124:0/1018273991 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801074905 0 0) 0x7f54c40b49f0 con 0x7f54c4150940 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c3fff640 1 --2- 172.21.15.124:0/1018273991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c40b3390 0x7f54c415d200 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f54c4072840 tx=0x7f54b802de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ad83b1788fa490 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c37fe640 1 -- 172.21.15.124:0/1018273991 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b802ec30 con 0x7f54c40b3390 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c37fe640 1 -- 172.21.15.124:0/1018273991 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8032070 con 0x7f54c40b3390 2023-12-13T23:56:40.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.082+0000 7f54c37fe640 1 -- 172.21.15.124:0/1018273991 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8038400 con 0x7f54c40b3390 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 -- 172.21.15.124:0/1018273991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c40b3390 msgr2=0x7f54c415d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 --2- 172.21.15.124:0/1018273991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c40b3390 0x7f54c415d200 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f54c4072840 tx=0x7f54b802de80 comp rx=0 tx=0).stop 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 -- 172.21.15.124:0/1018273991 shutdown_connections 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 --2- 172.21.15.124:0/1018273991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c40b3390 0x7f54c415d200 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 --2- 172.21.15.124:0/1018273991 >> v2:172.21.15.203:3300/0 conn(0x7f54c4150560 0x7f54c40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 -- 172.21.15.124:0/1018273991 >> 172.21.15.124:0/1018273991 conn(0x7f54c406a4d0 msgr2=0x7f54c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 -- 172.21.15.124:0/1018273991 shutdown_connections 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.083+0000 7f54caae5640 1 -- 172.21.15.124:0/1018273991 wait complete. 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.084+0000 7f54caae5640 1 Processor -- start 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.084+0000 7f54caae5640 1 -- start start 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 0x7f54c414d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c4158c30 0x7f54c41590e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c4159620 con 0x7f54c4150560 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c41f31d0 con 0x7f54c40b3390 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54caae5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c415b580 con 0x7f54c4158c30 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39024/0 (socket says 172.21.15.124:39024) 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 -- 172.21.15.124:0/589163770 learned_addr learned my addr 172.21.15.124:0/589163770 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c8881640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 0x7f54c414d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c9082640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c4158c30 0x7f54c41590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 -- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c4158c30 msgr2=0x7f54c41590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c4158c30 0x7f54c41590e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 -- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 msgr2=0x7f54c414d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 0x7f54c414d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c3fff640 1 -- 172.21.15.124:0/589163770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b800b6a0 con 0x7f54c4150560 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.085+0000 7f54c8881640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 0x7f54c414d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54c3fff640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f54b800bb30 tx=0x7f54b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b8038e00 con 0x7f54c4150560 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8041440 con 0x7f54c4150560 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c41f2c10 con 0x7f54c4150560 2023-12-13T23:56:40.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54b80417f0 con 0x7f54c4150560 2023-12-13T23:56:40.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.086+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c415af30 con 0x7f54c4150560 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.087+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c415b730 con 0x7f54c4150560 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.087+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f54b8041950 con 0x7f54c4150560 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.088+0000 7f54c17fa640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 0x7f54b006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.088+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f54b80ae4c0 con 0x7f54c4150560 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.088+0000 7f54c8881640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 0x7f54b006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.088+0000 7f54c8881640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 0x7f54b006d680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54b4006fd0 tx=0x7f54b4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.092+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f54c415b730 con 0x7f54c4150560 2023-12-13T23:56:40.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.258+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54c40677a0 con 0x7f54c4150560 2023-12-13T23:56:40.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.259+0000 7f54c17fa640 1 -- 172.21.15.124:0/589163770 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54b80790f0 con 0x7f54c4150560 2023-12-13T23:56:40.258 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:40.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 msgr2=0x7f54b006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 0x7f54b006d680 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f54b4006fd0 tx=0x7f54b4008040 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 msgr2=0x7f54c4148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f54b800bb30 tx=0x7f54b80046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 shutdown_connections 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f54b006b1c0 0x7f54b006d680 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f54c4158c30 0x7f54c41590e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54c4150560 0x7f54c4148800 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 --2- 172.21.15.124:0/589163770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f54c40b3390 0x7f54c414d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 >> 172.21.15.124:0/589163770 conn(0x7f54c406a4d0 msgr2=0x7f54c40cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 shutdown_connections 2023-12-13T23:56:40.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.260+0000 7f54caae5640 1 -- 172.21.15.124:0/589163770 wait complete. 2023-12-13T23:56:40.270 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:40.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.373+0000 7f07d37ce640 1 Processor -- start 2023-12-13T23:56:40.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 -- start start 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 0x7f07cc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc15e2a0 con 0x7f07cc15e730 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc15b520 con 0x7f07cc143490 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.374+0000 7f07d37ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc143870 con 0x7f07cc15abf0 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d156a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 0x7f07cc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d1d6b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d0d69640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f07cc15e730 0x7f07cc15a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:55130/0 (socket says 172.21.15.124:55130) 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d0d69640 1 -- 172.21.15.124:0/764074071 learned_addr learned my addr 172.21.15.124:0/764074071 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:40.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d1d6b640 1 -- 172.21.15.124:0/764074071 >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 msgr2=0x7f07cc0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d1d6b640 1 --2- 172.21.15.124:0/764074071 >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 0x7f07cc0b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d1d6b640 1 -- 172.21.15.124:0/764074071 >> v1:172.21.15.203:6789/0 conn(0x7f07cc15e730 legacy=0x7f07cc15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d1d6b640 1 -- 172.21.15.124:0/764074071 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc15b240 con 0x7f07cc15abf0 2023-12-13T23:56:40.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.375+0000 7f07d156a640 1 --2- 172.21.15.124:0/764074071 >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 0x7f07cc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.376+0000 7f07d1d6b640 1 --2- 172.21.15.124:0/764074071 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc0b1230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009d90 tx=0x7f07c402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6652662bbe1b7c46 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.376+0000 7f07c3fff640 1 -- 172.21.15.124:0/764074071 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c402cc50 con 0x7f07cc15abf0 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.376+0000 7f07c3fff640 1 -- 172.21.15.124:0/764074071 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c402cdb0 con 0x7f07cc15abf0 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.376+0000 7f07c3fff640 1 -- 172.21.15.124:0/764074071 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4034660 con 0x7f07cc15abf0 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 -- 172.21.15.124:0/764074071 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 msgr2=0x7f07cc0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 --2- 172.21.15.124:0/764074071 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc0b1230 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009d90 tx=0x7f07c402bea0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 -- 172.21.15.124:0/764074071 shutdown_connections 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 --2- 172.21.15.124:0/764074071 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc0b1230 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 --2- 172.21.15.124:0/764074071 >> v2:172.21.15.203:3300/0 conn(0x7f07cc143490 0x7f07cc0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 -- 172.21.15.124:0/764074071 >> 172.21.15.124:0/764074071 conn(0x7f07cc06a4d0 msgr2=0x7f07cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 -- 172.21.15.124:0/764074071 shutdown_connections 2023-12-13T23:56:40.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.377+0000 7f07d37ce640 1 -- 172.21.15.124:0/764074071 wait complete. 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.378+0000 7f07d37ce640 1 Processor -- start 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.378+0000 7f07d37ce640 1 -- start start 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07cc143490 0x7f07cc0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc15b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc12e540 con 0x7f07cc15e730 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc0ae540 con 0x7f07cc143490 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc12e260 con 0x7f07cc15abf0 2023-12-13T23:56:40.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:39034/0 (socket says 172.21.15.124:39034) 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 -- 172.21.15.124:0/3702188337 learned_addr learned my addr 172.21.15.124:0/3702188337 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d0d69640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc15b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d156a640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07cc143490 0x7f07cc0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 -- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 msgr2=0x7f07cc15b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc15b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 -- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07cc143490 msgr2=0x7f07cc0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07cc143490 0x7f07cc0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 -- 172.21.15.124:0/3702188337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc009400 con 0x7f07cc15e730 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d1d6b640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009d90 tx=0x7f07c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4040650 con 0x7f07cc15e730 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c40407b0 con 0x7f07cc15e730 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c4009a70 con 0x7f07cc15e730 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.379+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4040b60 con 0x7f07cc15e730 2023-12-13T23:56:40.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.380+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc1f39a0 con 0x7f07cc15e730 2023-12-13T23:56:40.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.381+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc1f3520 con 0x7f07cc15e730 2023-12-13T23:56:40.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.381+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f07c4040cc0 con 0x7f07cc15e730 2023-12-13T23:56:40.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.381+0000 7f07c1ffb640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 0x7f07b006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.381+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f07c40ac720 con 0x7f07cc15e730 2023-12-13T23:56:40.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.382+0000 7f07d156a640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 0x7f07b006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.382+0000 7f07d156a640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 0x7f07b006d680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009cd0 tx=0x7f07bc009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.386+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f07cc1f3520 con 0x7f07cc15e730 2023-12-13T23:56:40.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.550+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07cc1f3520 con 0x7f07cc15e730 2023-12-13T23:56:40.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.551+0000 7f07c1ffb640 1 -- 172.21.15.124:0/3702188337 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7f07cc1f3520 con 0x7f07cc15e730 2023-12-13T23:56:40.550 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 msgr2=0x7f07b006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 0x7f07b006d680 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009cd0 tx=0x7f07bc009190 comp rx=0 tx=0).stop 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 msgr2=0x7f07cc15bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f07c4009d90 tx=0x7f07c4004a40 comp rx=0 tx=0).stop 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 shutdown_connections 2023-12-13T23:56:40.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f07b006b1c0 0x7f07b006d680 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07cc15e730 0x7f07cc15bc60 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07cc15abf0 0x7f07cc15b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 --2- 172.21.15.124:0/3702188337 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f07cc143490 0x7f07cc0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 >> 172.21.15.124:0/3702188337 conn(0x7f07cc06a4d0 msgr2=0x7f07cc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 shutdown_connections 2023-12-13T23:56:40.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.553+0000 7f07d37ce640 1 -- 172.21.15.124:0/3702188337 wait complete. 2023-12-13T23:56:40.562 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:40.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.666+0000 7fc233c9b640 1 Processor -- start 2023-12-13T23:56:40.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.666+0000 7fc233c9b640 1 -- start start 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc233c9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc22c1409d0 0x7fc22c1618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc233c9b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c1406f0 con 0x7fc22c140310 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc233c9b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc22c143960 con 0x7fc22c06abe0 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc233c9b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc22c161e20 con 0x7fc22c1409d0 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc231a37640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc22c140310 0x7fc22c06a4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42348/0 (socket says 172.21.15.124:42348) 2023-12-13T23:56:40.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc231a37640 1 -- 172.21.15.124:0/1368368835 learned_addr learned my addr 172.21.15.124:0/1368368835 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc232238640 1 --2- 172.21.15.124:0/1368368835 >> v2:172.21.15.124:3300/0 conn(0x7fc22c1409d0 0x7fc22c1618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc230a35640 1 -- 172.21.15.124:0/1368368835 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484516572 0 0) 0x7fc22c1406f0 con 0x7fc22c140310 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc230a35640 1 -- 172.21.15.124:0/1368368835 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc218003520 con 0x7fc22c140310 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc232238640 1 -- 172.21.15.124:0/1368368835 >> v1:172.21.15.203:6789/0 conn(0x7fc22c06abe0 legacy=0x7fc22c06efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc232238640 1 -- 172.21.15.124:0/1368368835 >> v1:172.21.15.124:6789/0 conn(0x7fc22c140310 legacy=0x7fc22c06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.667+0000 7fc232238640 1 -- 172.21.15.124:0/1368368835 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c142de0 con 0x7fc22c1409d0 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.668+0000 7fc232238640 1 --2- 172.21.15.124:0/1368368835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c1409d0 0x7fc22c1618e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc22402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483bb77ff02adbb1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.668+0000 7fc230a35640 1 -- 172.21.15.124:0/1368368835 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22402cc60 con 0x7fc22c1409d0 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.668+0000 7fc230a35640 1 -- 172.21.15.124:0/1368368835 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22402cdc0 con 0x7fc22c1409d0 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.668+0000 7fc230a35640 1 -- 172.21.15.124:0/1368368835 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc224034660 con 0x7fc22c1409d0 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 -- 172.21.15.124:0/1368368835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c1409d0 msgr2=0x7fc22c1618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 --2- 172.21.15.124:0/1368368835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c1409d0 0x7fc22c1618e0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fc224009d90 tx=0x7fc22402beb0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 -- 172.21.15.124:0/1368368835 shutdown_connections 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 --2- 172.21.15.124:0/1368368835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c1409d0 0x7fc22c1618e0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 -- 172.21.15.124:0/1368368835 >> 172.21.15.124:0/1368368835 conn(0x7fc22c0b1e50 msgr2=0x7fc22c0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 -- 172.21.15.124:0/1368368835 shutdown_connections 2023-12-13T23:56:40.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.669+0000 7fc233c9b640 1 -- 172.21.15.124:0/1368368835 wait complete. 2023-12-13T23:56:40.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.670+0000 7fc233c9b640 1 Processor -- start 2023-12-13T23:56:40.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.670+0000 7fc233c9b640 1 -- start start 2023-12-13T23:56:40.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc22c06abe0 0x7fc22c13f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc22c1409d0 0x7fc22c1f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c1458f0 con 0x7fc22c140310 2023-12-13T23:56:40.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c1f6da0 con 0x7fc22c1409d0 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc22c1f7080 con 0x7fc22c06abe0 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37038/0 (socket says 172.21.15.124:37038) 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 -- 172.21.15.124:0/4167420979 learned_addr learned my addr 172.21.15.124:0/4167420979 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231a37640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc22c06abe0 0x7fc22c13f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc232238640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc22c1409d0 0x7fc22c1f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 -- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc22c06abe0 msgr2=0x7fc22c13f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc22c06abe0 0x7fc22c13f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 -- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc22c1409d0 msgr2=0x7fc22c1f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc22c1409d0 0x7fc22c1f62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 -- 172.21.15.124:0/4167420979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc224009a70 con 0x7fc22c140310 2023-12-13T23:56:40.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc231236640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c400 tx=0x7fc21c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c0052d0 con 0x7fc22c140310 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c005430 con 0x7fc22c140310 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22c145b90 con 0x7fc22c140310 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c003830 con 0x7fc22c140310 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.671+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc22c13fe30 con 0x7fc22c140310 2023-12-13T23:56:40.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.673+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc22c1460b0 con 0x7fc22c140310 2023-12-13T23:56:40.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.673+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc21c003990 con 0x7fc22c140310 2023-12-13T23:56:40.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.673+0000 7fc2227fc640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 0x7fc21006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.673+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fc21c089040 con 0x7fc22c140310 2023-12-13T23:56:40.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.673+0000 7fc231a37640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 0x7fc21006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.674+0000 7fc231a37640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 0x7fc21006d6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2280011e0 tx=0x7fc228009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.677+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc22c1460b0 con 0x7fc22c140310 2023-12-13T23:56:40.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.845+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc22c1460b0 con 0x7fc22c140310 2023-12-13T23:56:40.844 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.845+0000 7fc2227fc640 1 -- 172.21.15.124:0/4167420979 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc22c1460b0 con 0x7fc22c140310 2023-12-13T23:56:40.844 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:40.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 msgr2=0x7fc21006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 0x7fc21006d6b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc2280011e0 tx=0x7fc228009040 comp rx=0 tx=0).stop 2023-12-13T23:56:40.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 msgr2=0x7fc22c146390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.846 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00c400 tx=0x7fc21c00c8c0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 shutdown_connections 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21006b1f0 0x7fc21006d6b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc22c1409d0 0x7fc22c1f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc22c140310 0x7fc22c146390 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 --2- 172.21.15.124:0/4167420979 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc22c06abe0 0x7fc22c13f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 >> 172.21.15.124:0/4167420979 conn(0x7fc22c0b1e50 msgr2=0x7fc22c15ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 shutdown_connections 2023-12-13T23:56:40.847 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.847+0000 7fc233c9b640 1 -- 172.21.15.124:0/4167420979 wait complete. 2023-12-13T23:56:40.857 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:40.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.961+0000 7f41dd97c640 1 Processor -- start 2023-12-13T23:56:40.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.961+0000 7f41dd97c640 1 -- start start 2023-12-13T23:56:40.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41dd97c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41d80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41dd97c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f41d80d2d90 0x7f41d80b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41dd97c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d80b2b10 con 0x7f41d80d2910 2023-12-13T23:56:40.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41dd97c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80d1590 con 0x7f41d80d2d90 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41dd97c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80d1870 con 0x7f41d815d330 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41ce7fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f41d80d2910 0x7f41d80d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42354/0 (socket says 172.21.15.124:42354) 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41ce7fc640 1 -- 172.21.15.124:0/2944524261 learned_addr learned my addr 172.21.15.124:0/2944524261 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d77fe640 1 --2- 172.21.15.124:0/2944524261 >> v2:172.21.15.203:3300/0 conn(0x7f41d80d2d90 0x7f41d80b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951009648 0 0) 0x7f41d80b2b10 con 0x7f41d80d2910 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41d80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41c4003520 con 0x7f41d80d2910 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3569111744 0 0) 0x7f41c4003520 con 0x7f41d80d2910 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 msgr2=0x7f41d80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41d80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 >> v2:172.21.15.203:3300/0 conn(0x7f41d80d2d90 msgr2=0x7f41d80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 --2- 172.21.15.124:0/2944524261 >> v2:172.21.15.203:3300/0 conn(0x7f41d80d2d90 0x7f41d80b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80d1b50 con 0x7f41d80d2910 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f41c0001670 con 0x7f41d80d2910 2023-12-13T23:56:40.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 >> v1:172.21.15.124:6789/0 conn(0x7f41d80d2910 legacy=0x7f41d80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41c4003970 0x7f41c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c40047c0 0x7f41c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80d1590 con 0x7f41c4003970 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80d1870 con 0x7f41c40047c0 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c400b980 con 0x7f41d815d330 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.962+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41c4003970 0x7f41c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41ce7fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c40047c0 0x7f41c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d77fe640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 -- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 msgr2=0x7f41c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41c4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 -- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c40047c0 msgr2=0x7f41c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:40.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c40047c0 0x7f41c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 -- 172.21.15.124:0/2944524261 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c4003520 con 0x7f41c4003970 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41ce7fc640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41c40047c0 0x7f41c4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d77fe640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41d815d330 0x7f41c4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.963+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41c4003970 0x7f41c4003d90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009af0 tx=0x7f41c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ce345cbe670fda server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c802c6b0 con 0x7f41c4003970 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c802c810 con 0x7f41c4003970 2023-12-13T23:56:40.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41d67fc640 1 -- 172.21.15.124:0/2944524261 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c802cba0 con 0x7f41c4003970 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 -- 172.21.15.124:0/2944524261 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41c4003970 msgr2=0x7f41c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 --2- 172.21.15.124:0/2944524261 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41c4003970 0x7f41c4003d90 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f41c8009af0 tx=0x7f41c40057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41d77fe640 1 -- 172.21.15.124:0/2944524261 reap_dead start 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 -- 172.21.15.124:0/2944524261 shutdown_connections 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 -- 172.21.15.124:0/2944524261 >> 172.21.15.124:0/2944524261 conn(0x7f41d8072020 msgr2=0x7f41d8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 -- 172.21.15.124:0/2944524261 shutdown_connections 2023-12-13T23:56:40.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.964+0000 7f41dd97c640 1 -- 172.21.15.124:0/2944524261 wait complete. 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.965+0000 7f41dd97c640 1 Processor -- start 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.965+0000 7f41dd97c640 1 -- start start 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 0x7f41d81f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 0x7f41d80b6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80b74a0 con 0x7f41d81f2c10 2023-12-13T23:56:40.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d80aa940 con 0x7f41d80b6ae0 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41dd97c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d81f3e20 con 0x7f41c4003970 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37054/0 (socket says 172.21.15.124:37054) 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 -- 172.21.15.124:0/784953817 learned_addr learned my addr 172.21.15.124:0/784953817 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41d77fe640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 0x7f41d80b6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 0x7f41d81f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 -- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 msgr2=0x7f41d81f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 0x7f41d81f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 -- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 msgr2=0x7f41d80b6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 0x7f41d80b6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41ce7fc640 1 -- 172.21.15.124:0/784953817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d81f71a0 con 0x7f41d81f2c10 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41d77fe640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 0x7f41d80b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.966+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 0x7f41d81f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41ce7fc640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f41c000e810 tx=0x7f41c000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c000c870 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c000c9d0 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d81f0650 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c000cd60 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.967+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d81f3470 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.968+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d8069400 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.968+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f41c001d460 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.969+0000 7f41d4ff9640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 0x7f41b806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.969+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f41c0089b70 con 0x7f41d81f2c10 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.969+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 0x7f41b806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.969+0000 7f41d6ffd640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 0x7f41b806d680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41c8036000 tx=0x7f41c8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:40.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:40.973+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f41c0054710 con 0x7f41d81f2c10 2023-12-13T23:56:41.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.140+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f41d80b58d0 con 0x7f41d81f2c10 2023-12-13T23:56:41.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.140+0000 7f41d4ff9640 1 -- 172.21.15.124:0/784953817 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f41c0057d30 con 0x7f41d81f2c10 2023-12-13T23:56:41.139 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 msgr2=0x7f41b806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 0x7f41b806d680 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41c8036000 tx=0x7f41c8030000 comp rx=0 tx=0).stop 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 msgr2=0x7f41d81f0110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f41c000e810 tx=0x7f41c000ecd0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 shutdown_connections 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f41b806b1c0 0x7f41b806d680 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f41d80b6ae0 0x7f41d80b6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41d81f2c10 0x7f41d81f0110 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 --2- 172.21.15.124:0/784953817 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f41c4003970 0x7f41d81f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 >> 172.21.15.124:0/784953817 conn(0x7f41d8072020 msgr2=0x7f41d815c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 shutdown_connections 2023-12-13T23:56:41.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.142+0000 7f41dd97c640 1 -- 172.21.15.124:0/784953817 wait complete. 2023-12-13T23:56:41.152 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:41.253 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.254+0000 7f02675c9640 1 Processor -- start 2023-12-13T23:56:41.253 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.254+0000 7f02675c9640 1 -- start start 2023-12-13T23:56:41.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f02675c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02600b1400 0x7f026015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f02675c9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f026015e730 0x7f026015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f02675c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026015e2a0 con 0x7f0260143490 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f02675c9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026015b520 con 0x7f026015e730 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f02675c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0260143870 con 0x7f02600b1400 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f0265b66640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f026015e730 0x7f026015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f0265b66640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f026015e730 0x7f026015aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37060/0 (socket says 172.21.15.124:37060) 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.255+0000 7f0265b66640 1 -- 172.21.15.124:0/2835660305 learned_addr learned my addr 172.21.15.124:0/2835660305 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0264b64640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02600b1400 0x7f026015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0265b66640 1 -- 172.21.15.124:0/2835660305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02600b1400 msgr2=0x7f026015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0265b66640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02600b1400 0x7f026015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0265b66640 1 -- 172.21.15.124:0/2835660305 >> v1:172.21.15.203:6789/0 conn(0x7f0260143490 legacy=0x7f02600b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0265b66640 1 -- 172.21.15.124:0/2835660305 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026015b240 con 0x7f026015e730 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0265b66640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f026015e730 0x7f026015aa20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0258009870 tx=0x7f025802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdd323bc6a89f0d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.256+0000 7f0257fff640 1 -- 172.21.15.124:0/2835660305 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258031640 con 0x7f026015e730 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f0257fff640 1 -- 172.21.15.124:0/2835660305 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02580317a0 con 0x7f026015e730 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f0257fff640 1 -- 172.21.15.124:0/2835660305 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258031b30 con 0x7f026015e730 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 -- 172.21.15.124:0/2835660305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f026015e730 msgr2=0x7f026015aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f026015e730 0x7f026015aa20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f0258009870 tx=0x7f025802b8f0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 -- 172.21.15.124:0/2835660305 shutdown_connections 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02600b1400 0x7f026015a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 --2- 172.21.15.124:0/2835660305 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f026015e730 0x7f026015aa20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 -- 172.21.15.124:0/2835660305 >> 172.21.15.124:0/2835660305 conn(0x7f026006a4d0 msgr2=0x7f026006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 -- 172.21.15.124:0/2835660305 shutdown_connections 2023-12-13T23:56:41.257 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.257+0000 7f02675c9640 1 -- 172.21.15.124:0/2835660305 wait complete. 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.258+0000 7f02675c9640 1 Processor -- start 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.258+0000 7f02675c9640 1 -- start start 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0260143490 0x7f026013f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026015e730 0x7f0260158150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02601409f0 con 0x7f02600b1400 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026013baa0 con 0x7f026015e730 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f02675c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026013fd70 con 0x7f0260143490 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37074/0 (socket says 172.21.15.124:37074) 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 -- 172.21.15.124:0/2326868653 learned_addr learned my addr 172.21.15.124:0/2326868653 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:41.260 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0264b64640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0260143490 0x7f026013f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265b66640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026015e730 0x7f0260158150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 -- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0260143490 msgr2=0x7f026013f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0260143490 0x7f026013f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 -- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026015e730 msgr2=0x7f0260158150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026015e730 0x7f0260158150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.259+0000 7f0265365640 1 -- 172.21.15.124:0/2326868653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02580094d0 con 0x7f02600b1400 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f0265365640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c7c0 tx=0x7f025c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c00aa50 con 0x7f02600b1400 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c00abb0 con 0x7f02600b1400 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0260140050 con 0x7f02600b1400 2023-12-13T23:56:41.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025c012420 con 0x7f02600b1400 2023-12-13T23:56:41.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.260+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0260140610 con 0x7f02600b1400 2023-12-13T23:56:41.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.261+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0260069400 con 0x7f02600b1400 2023-12-13T23:56:41.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.261+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f025c012580 con 0x7f02600b1400 2023-12-13T23:56:41.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.262+0000 7f0255ffb640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 0x7f024406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.262+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f025c089df0 con 0x7f02600b1400 2023-12-13T23:56:41.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.262+0000 7f0264b64640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 0x7f024406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.262+0000 7f0264b64640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 0x7f024406d680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f026015a0e0 tx=0x7f0250008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.266+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f025c054990 con 0x7f02600b1400 2023-12-13T23:56:41.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.446+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f02600b42f0 con 0x7f02600b1400 2023-12-13T23:56:41.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.446+0000 7f0255ffb640 1 -- 172.21.15.124:0/2326868653 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7f025c016080 con 0x7f02600b1400 2023-12-13T23:56:41.445 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:41.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 msgr2=0x7f024406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 0x7f024406d680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f026015a0e0 tx=0x7f0250008040 comp rx=0 tx=0).stop 2023-12-13T23:56:41.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 msgr2=0x7f026013b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f025c00c7c0 tx=0x7f025c00cc80 comp rx=0 tx=0).stop 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 shutdown_connections 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f024406b1c0 0x7f024406d680 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f026015e730 0x7f0260158150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0260143490 0x7f026013f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 --2- 172.21.15.124:0/2326868653 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02600b1400 0x7f026013b560 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 >> 172.21.15.124:0/2326868653 conn(0x7f026006a4d0 msgr2=0x7f02600af0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 shutdown_connections 2023-12-13T23:56:41.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.448+0000 7f02675c9640 1 -- 172.21.15.124:0/2326868653 wait complete. 2023-12-13T23:56:41.458 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:41.458 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:41.473 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:41.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.614+0000 7f306ef68640 1 Processor -- start 2023-12-13T23:56:41.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.614+0000 7f306ef68640 1 -- start start 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306ef68640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f30680ae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306ef68640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680adbb0 con 0x7f3068158ba0 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306ef68640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680b4520 con 0x7f3068071c80 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306ef68640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30680b3f60 con 0x7f30680ac370 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306cd04640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3068158ba0 0x7f306815c160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42356/0 (socket says 172.21.15.124:42356) 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f306cd04640 1 -- 172.21.15.124:0/4281470780 learned_addr learned my addr 172.21.15.124:0/4281470780 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f3067fff640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f30680ae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509574010 0 0) 0x7f30680adbb0 con 0x7f3068158ba0 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3058003520 con 0x7f3068158ba0 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666896402 0 0) 0x7f3058003520 con 0x7f3068158ba0 2023-12-13T23:56:41.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 msgr2=0x7f30680ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f30680ae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 >> v1:172.21.15.203:6789/0 conn(0x7f3068071c80 legacy=0x7f30680b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680b4240 con 0x7f3068158ba0 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f305c001670 con 0x7f3068158ba0 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 >> v1:172.21.15.124:6789/0 conn(0x7f3068158ba0 legacy=0x7f306815c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580039a0 0x7f3058003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.615+0000 7f30677fe640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3058004350 0x7f3058004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f30677fe640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f3058005120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30680b3f60 con 0x7f30580039a0 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305800ba30 con 0x7f3058004350 2023-12-13T23:56:41.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f305800bbf0 con 0x7f30680ac370 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580039a0 0x7f3058003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f3067fff640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3058004350 0x7f3058004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306d505640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f3058005120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 -- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 msgr2=0x7f3058005120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f3058005120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 -- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3058004350 msgr2=0x7f3058004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3058004350 0x7f3058004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306cd04640 1 -- 172.21.15.124:0/4281470780 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3058003520 con 0x7f30580039a0 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.616+0000 7f306d505640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f30680ac370 0x7f3058005120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306cd04640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580039a0 0x7f3058003d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c190 tx=0x7f30580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8357079d80101e4b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c005ad0 con 0x7f30580039a0 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c005c50 con 0x7f30580039a0 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f30677fe640 1 -- 172.21.15.124:0/4281470780 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c02c850 con 0x7f30580039a0 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 -- 172.21.15.124:0/4281470780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580039a0 msgr2=0x7f3058003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 --2- 172.21.15.124:0/4281470780 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580039a0 0x7f3058003d80 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c190 tx=0x7f30580057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306d505640 1 -- 172.21.15.124:0/4281470780 reap_dead start 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 -- 172.21.15.124:0/4281470780 shutdown_connections 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 -- 172.21.15.124:0/4281470780 >> 172.21.15.124:0/4281470780 conn(0x7f30680b4720 msgr2=0x7f30680b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 -- 172.21.15.124:0/4281470780 shutdown_connections 2023-12-13T23:56:41.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.617+0000 7f306ef68640 1 -- 172.21.15.124:0/4281470780 wait complete. 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.618+0000 7f306ef68640 1 Processor -- start 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.618+0000 7f306ef68640 1 -- start start 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30580039a0 0x7f306814d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f306813f550 0x7f306813f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30681555c0 con 0x7f3068155140 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306813ff10 con 0x7f30580039a0 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f306ef68640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30680aecd0 con 0x7f306813f550 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f3067fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f3067fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37084/0 (socket says 172.21.15.124:37084) 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.619+0000 7f3067fff640 1 -- 172.21.15.124:0/264691022 learned_addr learned my addr 172.21.15.124:0/264691022 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f306cd04640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30580039a0 0x7f306814d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f306d505640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f306813f550 0x7f306813f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 -- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f306813f550 msgr2=0x7f306813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f306813f550 0x7f306813f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 -- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30580039a0 msgr2=0x7f306814d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30580039a0 0x7f306814d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 -- 172.21.15.124:0/264691022 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3068155840 con 0x7f3068155140 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f3067fff640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f30540119a0 tx=0x7f3054011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400faf0 con 0x7f3068155140 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305400fc50 con 0x7f3068155140 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068155b20 con 0x7f3068155140 2023-12-13T23:56:41.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054014440 con 0x7f3068155140 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.620+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30681560e0 con 0x7f3068155140 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.621+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3068069400 con 0x7f3068155140 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.621+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f30540145a0 con 0x7f3068155140 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.622+0000 7f30657fa640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 0x7f304c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.622+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f305408a3d0 con 0x7f3068155140 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.622+0000 7f306cd04640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 0x7f304c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.623+0000 7f306cd04640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 0x7f304c06d6b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f305c034000 tx=0x7f305c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.626+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3054054ef0 con 0x7f3068155140 2023-12-13T23:56:41.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.797+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30680b34c0 con 0x7f3068155140 2023-12-13T23:56:41.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.798+0000 7f30657fa640 1 -- 172.21.15.124:0/264691022 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f3054058510 con 0x7f3068155140 2023-12-13T23:56:41.798 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:41.798 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 msgr2=0x7f304c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 0x7f304c06d6b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f305c034000 tx=0x7f305c009030 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 msgr2=0x7f30680ae790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f30540119a0 tx=0x7f3054011e60 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 shutdown_connections 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f304c06b1f0 0x7f304c06d6b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f306813f550 0x7f306813f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3068155140 0x7f30680ae790 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.800+0000 7f306ef68640 1 --2- 172.21.15.124:0/264691022 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f30580039a0 0x7f306814d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.801+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 >> 172.21.15.124:0/264691022 conn(0x7f30680b4720 msgr2=0x7f306815c770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.801+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 shutdown_connections 2023-12-13T23:56:41.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.801+0000 7f306ef68640 1 -- 172.21.15.124:0/264691022 wait complete. 2023-12-13T23:56:41.813 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:41.813 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:41.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.917+0000 7fdd367a9640 1 Processor -- start 2023-12-13T23:56:41.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.917+0000 7fdd367a9640 1 -- start start 2023-12-13T23:56:41.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd367a9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdd30143490 0x7fdd300b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd367a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 0x7fdd300b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd367a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd3015e2a0 con 0x7fdd3015a4e0 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd367a9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3015b520 con 0x7fdd30143490 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd367a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd30143870 con 0x7fdd3015e730 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd2ffff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdd30143490 0x7fdd300b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd34d46640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdd3015a4e0 0x7fdd3015a930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42358/0 (socket says 172.21.15.124:42358) 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd34d46640 1 -- 172.21.15.124:0/2510325986 learned_addr learned my addr 172.21.15.124:0/2510325986 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.918+0000 7fdd2f7fe640 1 --2- 172.21.15.124:0/2510325986 >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 0x7fdd300b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2ffff640 1 -- 172.21.15.124:0/2510325986 >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 msgr2=0x7fdd300b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2ffff640 1 --2- 172.21.15.124:0/2510325986 >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 0x7fdd300b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2ffff640 1 -- 172.21.15.124:0/2510325986 >> v1:172.21.15.124:6789/0 conn(0x7fdd3015a4e0 legacy=0x7fdd3015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2ffff640 1 -- 172.21.15.124:0/2510325986 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3015b240 con 0x7fdd30143490 2023-12-13T23:56:41.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2f7fe640 1 --2- 172.21.15.124:0/2510325986 >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 0x7fdd300b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2ffff640 1 --2- 172.21.15.124:0/2510325986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd30143490 0x7fdd300b0cf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdd300aeca0 tx=0x7fdd2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33b5101899be38f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2effd640 1 -- 172.21.15.124:0/2510325986 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20031830 con 0x7fdd30143490 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2effd640 1 -- 172.21.15.124:0/2510325986 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20031990 con 0x7fdd30143490 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.919+0000 7fdd2effd640 1 -- 172.21.15.124:0/2510325986 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd20031d20 con 0x7fdd30143490 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 -- 172.21.15.124:0/2510325986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd30143490 msgr2=0x7fdd300b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2510325986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd30143490 0x7fdd300b0cf0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fdd300aeca0 tx=0x7fdd2002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 -- 172.21.15.124:0/2510325986 shutdown_connections 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2510325986 >> v2:172.21.15.203:3300/0 conn(0x7fdd3015e730 0x7fdd300b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2510325986 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd30143490 0x7fdd300b0cf0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 -- 172.21.15.124:0/2510325986 >> 172.21.15.124:0/2510325986 conn(0x7fdd3006a4d0 msgr2=0x7fdd3006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:41.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 -- 172.21.15.124:0/2510325986 shutdown_connections 2023-12-13T23:56:41.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.920+0000 7fdd367a9640 1 -- 172.21.15.124:0/2510325986 wait complete. 2023-12-13T23:56:41.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.921+0000 7fdd367a9640 1 Processor -- start 2023-12-13T23:56:41.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.921+0000 7fdd367a9640 1 -- start start 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd30143490 0x7fdd3013d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 0x7fdd30159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3013a130 con 0x7fdd3015e730 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3013da10 con 0x7fdd30143490 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd367a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd30139800 con 0x7fdd3015a4e0 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd34d46640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd34d46640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37098/0 (socket says 172.21.15.124:37098) 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd34d46640 1 -- 172.21.15.124:0/2002448057 learned_addr learned my addr 172.21.15.124:0/2002448057 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd2ffff640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd30143490 0x7fdd3013d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd2f7fe640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 0x7fdd30159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.922+0000 7fdd34d46640 1 -- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 msgr2=0x7fdd30159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd34d46640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 0x7fdd30159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd34d46640 1 -- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd30143490 msgr2=0x7fdd3013d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd34d46640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd30143490 0x7fdd3013d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd34d46640 1 -- 172.21.15.124:0/2002448057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20009400 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd2f7fe640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 0x7fdd30159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd34d46640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800cbc0 tx=0x7fdd2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800ad40 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd28012400 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2800aea0 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd30139ae0 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.923+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd300ae230 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.924+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3013a2e0 con 0x7fdd3015e730 2023-12-13T23:56:41.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.924+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdd28007070 con 0x7fdd3015e730 2023-12-13T23:56:41.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.924+0000 7fdd2cff9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 0x7fdd1c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:41.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.925+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fdd280899b0 con 0x7fdd3015e730 2023-12-13T23:56:41.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.925+0000 7fdd2ffff640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 0x7fdd1c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:41.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.925+0000 7fdd2ffff640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 0x7fdd1c06d680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd2003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:41.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:41.929+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdd3013a2e0 con 0x7fdd3015e730 2023-12-13T23:56:41.947 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:56:41.949 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:56:42.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.100+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd30069400 con 0x7fdd3015e730 2023-12-13T23:56:42.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.100+0000 7fdd2cff9640 1 -- 172.21.15.124:0/2002448057 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd28017020 con 0x7fdd3015e730 2023-12-13T23:56:42.099 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:42.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 msgr2=0x7fdd1c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 0x7fdd1c06d680 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fdd20036000 tx=0x7fdd2003c040 comp rx=0 tx=0).stop 2023-12-13T23:56:42.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 msgr2=0x7fdd3013ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800cbc0 tx=0x7fdd2800a430 comp rx=0 tx=0).stop 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 shutdown_connections 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdd1c06b1c0 0x7fdd1c06d680 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd3015e730 0x7fdd3013ffa0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdd3015a4e0 0x7fdd30159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 --2- 172.21.15.124:0/2002448057 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdd30143490 0x7fdd3013d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 >> 172.21.15.124:0/2002448057 conn(0x7fdd3006a4d0 msgr2=0x7fdd3015b6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 shutdown_connections 2023-12-13T23:56:42.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.102+0000 7fdd367a9640 1 -- 172.21.15.124:0/2002448057 wait complete. 2023-12-13T23:56:42.112 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:42.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.216+0000 7f1929d41640 1 Processor -- start 2023-12-13T23:56:42.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.216+0000 7f1929d41640 1 -- start start 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1929d41640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19240cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1929d41640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1924071770 con 0x7f19240b4370 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1929d41640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19240ccdc0 con 0x7f1924144980 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1929d41640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192415ced0 con 0x7f19240ceb70 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1923fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f19240b4370 0x7f1924071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42362/0 (socket says 172.21.15.124:42362) 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1923fff640 1 -- 172.21.15.124:0/1015445225 learned_addr learned my addr 172.21.15.124:0/1015445225 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:42.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1921539158 0 0) 0x7f1924071770 con 0x7f19240b4370 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1910003520 con 0x7f19240b4370 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f191bfff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19240cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043698053 0 0) 0x7f1910003520 con 0x7f19240b4370 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 msgr2=0x7f19240cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19240cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 >> v1:172.21.15.203:6789/0 conn(0x7f1924144980 legacy=0x7f192415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19240cd0a0 con 0x7f19240b4370 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f191c0019f0 con 0x7f19240b4370 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 >> v1:172.21.15.124:6789/0 conn(0x7f19240b4370 legacy=0x7f1924071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19100039a0 0x7f1910003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1910004d00 0x7f1910005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192415ced0 con 0x7f1910004d00 2023-12-13T23:56:42.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f191000ba60 con 0x7f19100039a0 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191000bc20 con 0x7f19240ceb70 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.217+0000 7f1923fff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1910004d00 0x7f1910005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f191bfff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f19237fe640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19100039a0 0x7f1910003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 -- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 msgr2=0x7f19100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f19240ceb70 0x7f19100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 -- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19100039a0 msgr2=0x7f1910003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19100039a0 0x7f1910003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 -- 172.21.15.124:0/1015445225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910003520 con 0x7f1910004d00 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f19237fe640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19100039a0 0x7f1910003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.218+0000 7f1923fff640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1910004d00 0x7f1910005180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f191c002230 tx=0x7f19100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf12aedc4ae4326c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c034630 con 0x7f1910004d00 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f191c02f070 con 0x7f1910004d00 2023-12-13T23:56:42.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1922ffd640 1 -- 172.21.15.124:0/1015445225 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f191c034a10 con 0x7f1910004d00 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 -- 172.21.15.124:0/1015445225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1910004d00 msgr2=0x7f1910005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 --2- 172.21.15.124:0/1015445225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1910004d00 0x7f1910005180 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f191c002230 tx=0x7f19100057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1923fff640 1 -- 172.21.15.124:0/1015445225 reap_dead start 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 -- 172.21.15.124:0/1015445225 shutdown_connections 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 -- 172.21.15.124:0/1015445225 >> 172.21.15.124:0/1015445225 conn(0x7f192406a4d0 msgr2=0x7f192406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 -- 172.21.15.124:0/1015445225 shutdown_connections 2023-12-13T23:56:42.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.219+0000 7f1929d41640 1 -- 172.21.15.124:0/1015445225 wait complete. 2023-12-13T23:56:42.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.220+0000 7f1929d41640 1 Processor -- start 2023-12-13T23:56:42.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.220+0000 7f1929d41640 1 -- start start 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1924132740 0x7f192413def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19241561e0 0x7f19241f3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19241555c0 con 0x7f192413e430 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192413e8b0 con 0x7f19241561e0 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1929d41640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1924156660 con 0x7f1924132740 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37110/0 (socket says 172.21.15.124:37110) 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 -- 172.21.15.124:0/1828538934 learned_addr learned my addr 172.21.15.124:0/1828538934 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 -- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1924132740 msgr2=0x7f192413def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f1923fff640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19241561e0 0x7f19241f3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1924132740 0x7f192413def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 -- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19241561e0 msgr2=0x7f19241f3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19241561e0 0x7f19241f3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.221+0000 7f191bfff640 1 -- 172.21.15.124:0/1828538934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924157490 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f191bfff640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f190c011940 tx=0x7f190c011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c004240 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c0043a0 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c00fb20 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f192413be40 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.222+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1924156b60 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.223+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1924069400 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.223+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f190c00fc80 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.223+0000 7f1920ff9640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 0x7f190406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.224+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f190c08a200 con 0x7f192413e430 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.224+0000 7f19237fe640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 0x7f190406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.224+0000 7f19237fe640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 0x7f190406d6b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1914007fd0 tx=0x7f1914009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.228+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f190c054da0 con 0x7f192413e430 2023-12-13T23:56:42.392 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.393+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1924159390 con 0x7f192413e430 2023-12-13T23:56:42.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.393+0000 7f1920ff9640 1 -- 172.21.15.124:0/1828538934 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7f190c0583c0 con 0x7f192413e430 2023-12-13T23:56:42.393 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:42.394 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 msgr2=0x7f190406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 0x7f190406d6b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f1914007fd0 tx=0x7f1914009040 comp rx=0 tx=0).stop 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 msgr2=0x7f1924155ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f190c011940 tx=0x7f190c011e00 comp rx=0 tx=0).stop 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 shutdown_connections 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f190406b1f0 0x7f190406d6b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f19241561e0 0x7f19241f3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f192413e430 0x7f1924155ca0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 --2- 172.21.15.124:0/1828538934 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1924132740 0x7f192413def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 >> 172.21.15.124:0/1828538934 conn(0x7f192406a4d0 msgr2=0x7f19240cb250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 shutdown_connections 2023-12-13T23:56:42.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.395+0000 7f1929d41640 1 -- 172.21.15.124:0/1828538934 wait complete. 2023-12-13T23:56:42.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:42.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.509+0000 7fc130953640 1 Processor -- start 2023-12-13T23:56:42.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.509+0000 7fc130953640 1 -- start start 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc130953640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c0b1400 0x7fc12c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc130953640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc12c15e730 0x7fc12c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc130953640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c15e2a0 con 0x7fc12c143490 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc130953640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c15b520 con 0x7fc12c15e730 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc130953640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c143870 con 0x7fc12c0b1400 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc12ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc12c15e730 0x7fc12c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc12ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc12c15e730 0x7fc12c15aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37124/0 (socket says 172.21.15.124:37124) 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.510+0000 7fc12ad9d640 1 -- 172.21.15.124:0/3459733441 learned_addr learned my addr 172.21.15.124:0/3459733441 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc129d9b640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c0b1400 0x7fc12c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12ad9d640 1 -- 172.21.15.124:0/3459733441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c0b1400 msgr2=0x7fc12c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12ad9d640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c0b1400 0x7fc12c15a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12ad9d640 1 -- 172.21.15.124:0/3459733441 >> v1:172.21.15.203:6789/0 conn(0x7fc12c143490 legacy=0x7fc12c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12ad9d640 1 -- 172.21.15.124:0/3459733441 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c15b240 con 0x7fc12c15e730 2023-12-13T23:56:42.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12ad9d640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c15e730 0x7fc12c15aa20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc12400bb10 tx=0x7fc12402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebbc254c6b7cb18e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.511+0000 7fc12959a640 1 -- 172.21.15.124:0/3459733441 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124036440 con 0x7fc12c15e730 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc12959a640 1 -- 172.21.15.124:0/3459733441 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1240365a0 con 0x7fc12c15e730 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc12959a640 1 -- 172.21.15.124:0/3459733441 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124036930 con 0x7fc12c15e730 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 -- 172.21.15.124:0/3459733441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c15e730 msgr2=0x7fc12c15aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c15e730 0x7fc12c15aa20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc12400bb10 tx=0x7fc12402e150 comp rx=0 tx=0).stop 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 -- 172.21.15.124:0/3459733441 shutdown_connections 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c0b1400 0x7fc12c15a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 --2- 172.21.15.124:0/3459733441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c15e730 0x7fc12c15aa20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 -- 172.21.15.124:0/3459733441 >> 172.21.15.124:0/3459733441 conn(0x7fc12c06a4d0 msgr2=0x7fc12c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 -- 172.21.15.124:0/3459733441 shutdown_connections 2023-12-13T23:56:42.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.512+0000 7fc130953640 1 -- 172.21.15.124:0/3459733441 wait complete. 2023-12-13T23:56:42.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.513+0000 7fc130953640 1 Processor -- start 2023-12-13T23:56:42.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 -- start start 2023-12-13T23:56:42.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 0x7fc12c159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c15e730 0x7fc12c0cd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1f71a0 con 0x7fc12c143490 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c159ec0 con 0x7fc12c0b1400 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc130953640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c1f6520 con 0x7fc12c15e730 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc129d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc129d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37134/0 (socket says 172.21.15.124:37134) 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc129d9b640 1 -- 172.21.15.124:0/1755055651 learned_addr learned my addr 172.21.15.124:0/1755055651 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.514+0000 7fc12a59c640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 0x7fc12c159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc12ad9d640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c15e730 0x7fc12c0cd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 -- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c15e730 msgr2=0x7fc12c0cd9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c15e730 0x7fc12c0cd9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 -- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 msgr2=0x7fc12c159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.519 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 0x7fc12c159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 -- 172.21.15.124:0/1755055651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12400b770 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc12a59c640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 0x7fc12c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc129d9b640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc12000d800 tx=0x7fc12000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000b6c0 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12000b840 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c1f6800 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12000bbf0 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.515+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c1f6dc0 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.516+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c1f72e0 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.516+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc12000bd50 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.517+0000 7fc11affd640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 0x7fc10c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.517+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fc12008a1a0 con 0x7fc12c143490 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.517+0000 7fc12a59c640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 0x7fc10c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.520 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.518+0000 7fc12a59c640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 0x7fc10c06d680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cb6b0 tx=0x7fc11c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.521+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc12c1f72e0 con 0x7fc12c143490 2023-12-13T23:56:42.684 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.685+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc12c1f72e0 con 0x7fc12c143490 2023-12-13T23:56:42.684 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.685+0000 7fc11affd640 1 -- 172.21.15.124:0/1755055651 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc12c1f72e0 con 0x7fc12c143490 2023-12-13T23:56:42.685 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 msgr2=0x7fc10c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 0x7fc10c06d680 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0cb6b0 tx=0x7fc11c009040 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 msgr2=0x7fc12c15b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc12000d800 tx=0x7fc12000dcc0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 shutdown_connections 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc10c06b1c0 0x7fc10c06d680 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc12c15e730 0x7fc12c0cd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc12c143490 0x7fc12c15b660 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 --2- 172.21.15.124:0/1755055651 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc12c0b1400 0x7fc12c159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 >> 172.21.15.124:0/1755055651 conn(0x7fc12c06a4d0 msgr2=0x7fc12c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.687 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 shutdown_connections 2023-12-13T23:56:42.687 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.687+0000 7fc130953640 1 -- 172.21.15.124:0/1755055651 wait complete. 2023-12-13T23:56:42.697 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:42.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.803+0000 7f2fae733640 1 Processor -- start 2023-12-13T23:56:42.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.803+0000 7f2fae733640 1 -- start start 2023-12-13T23:56:42.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.803+0000 7f2fae733640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fa8144a00 0x7f2fa806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.803+0000 7f2fae733640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fae733640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80b2ef0 con 0x7f2fa81415f0 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fae733640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80b3b70 con 0x7f2fa8144a00 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fae733640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa8142510 con 0x7f2fa8144440 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fa8144a00 0x7f2fa806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fa8144a00 0x7f2fa806b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37136/0 (socket says 172.21.15.124:37136) 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 -- 172.21.15.124:0/3414436762 learned_addr learned my addr 172.21.15.124:0/3414436762 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:42.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 -- 172.21.15.124:0/3414436762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 msgr2=0x7f2fa806ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 --2- 172.21.15.124:0/3414436762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 -- 172.21.15.124:0/3414436762 >> v1:172.21.15.203:6789/0 conn(0x7f2fa81415f0 legacy=0x7f2fa8141e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.804+0000 7f2fa77fe640 1 -- 172.21.15.124:0/3414436762 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80b31d0 con 0x7f2fa8144a00 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fa77fe640 1 --2- 172.21.15.124:0/3414436762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa806b4e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0098a0 tx=0x7f2f9c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b35919e470090e1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fa6ffd640 1 -- 172.21.15.124:0/3414436762 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c031640 con 0x7f2fa8144a00 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fa6ffd640 1 -- 172.21.15.124:0/3414436762 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c0317a0 con 0x7f2fa8144a00 2023-12-13T23:56:42.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fa6ffd640 1 -- 172.21.15.124:0/3414436762 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c031b30 con 0x7f2fa8144a00 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 -- 172.21.15.124:0/3414436762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 msgr2=0x7f2fa806b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 --2- 172.21.15.124:0/3414436762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa806b4e0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0098a0 tx=0x7f2f9c02b8f0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 -- 172.21.15.124:0/3414436762 shutdown_connections 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 --2- 172.21.15.124:0/3414436762 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa806ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 --2- 172.21.15.124:0/3414436762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa806b4e0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 -- 172.21.15.124:0/3414436762 >> 172.21.15.124:0/3414436762 conn(0x7f2fa8070e70 msgr2=0x7f2fa8071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 -- 172.21.15.124:0/3414436762 shutdown_connections 2023-12-13T23:56:42.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.805+0000 7f2fae733640 1 -- 172.21.15.124:0/3414436762 wait complete. 2023-12-13T23:56:42.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 Processor -- start 2023-12-13T23:56:42.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 -- start start 2023-12-13T23:56:42.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa81415f0 0x7f2fa80ae060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa80cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa80c93c0 con 0x7f2fa8144a00 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa806d0e0 con 0x7f2fa81415f0 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.807+0000 7f2fae733640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa806d330 con 0x7f2fa8144440 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37150/0 (socket says 172.21.15.124:37150) 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 -- 172.21.15.124:0/2382681991 learned_addr learned my addr 172.21.15.124:0/2382681991 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fa7fff640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa81415f0 0x7f2fa80ae060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fa77fe640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa80cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 -- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 msgr2=0x7f2fa80cb590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa80cb590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 -- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa81415f0 msgr2=0x7f2fa80ae060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa81415f0 0x7f2fa80ae060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 -- 172.21.15.124:0/2382681991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0094d0 con 0x7f2fa8144a00 2023-12-13T23:56:42.817 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2faccd0640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000b970 tx=0x7f2fa000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa000cb40 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa000ccc0 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa806d610 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa0014680 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.808+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa806dbd0 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.809+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa806ddc0 con 0x7f2fa8144a00 2023-12-13T23:56:42.818 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.810+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2fa001d3b0 con 0x7f2fa8144a00 2023-12-13T23:56:42.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.810+0000 7f2fa4ff9640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 0x7f2f9406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:42.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.810+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2fa00899a0 con 0x7f2fa8144a00 2023-12-13T23:56:42.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.810+0000 7f2fa7fff640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 0x7f2f9406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:42.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.811+0000 7f2fa7fff640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 0x7f2f9406d6b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2f98004f60 tx=0x7f2f98007e30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:42.819 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.814+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2fa0054680 con 0x7f2fa8144a00 2023-12-13T23:56:42.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.980+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f2fa8069400 con 0x7f2fa8144a00 2023-12-13T23:56:42.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.980+0000 7f2fa4ff9640 1 -- 172.21.15.124:0/2382681991 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f2fa0019020 con 0x7f2fa8144a00 2023-12-13T23:56:42.980 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:42.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 msgr2=0x7f2f9406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 0x7f2f9406d6b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2f98004f60 tx=0x7f2f98007e30 comp rx=0 tx=0).stop 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 msgr2=0x7f2fa80cbad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f2fa000b970 tx=0x7f2fa000be30 comp rx=0 tx=0).stop 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 shutdown_connections 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2f9406b1f0 0x7f2f9406d6b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fa8144a00 0x7f2fa80cbad0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fa8144440 0x7f2fa80cb590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 --2- 172.21.15.124:0/2382681991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fa81415f0 0x7f2fa80ae060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:42.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 >> 172.21.15.124:0/2382681991 conn(0x7f2fa8070e70 msgr2=0x7f2fa81434d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:42.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 shutdown_connections 2023-12-13T23:56:42.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:42.982+0000 7f2fae733640 1 -- 172.21.15.124:0/2382681991 wait complete. 2023-12-13T23:56:42.993 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:43.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 Processor -- start 2023-12-13T23:56:43.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 -- start start 2023-12-13T23:56:43.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbe415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe415e2a0 con 0x7ffbe40b1400 2023-12-13T23:56:43.107 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbe415b240 con 0x7ffbe4143490 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.097+0000 7ffbe9941640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe415b520 con 0x7ffbe415e730 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe27fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ffbe40b1400 0x7ffbe415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42374/0 (socket says 172.21.15.124:42374) 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3461231020 learned_addr learned my addr 172.21.15.124:0/3461231020 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390321448 0 0) 0x7ffbe415e2a0 con 0x7ffbe40b1400 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbcc005730 con 0x7ffbe40b1400 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe37fe640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbe415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 339457805 0 0) 0x7ffbcc005730 con 0x7ffbe40b1400 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 msgr2=0x7ffbe415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbe415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 >> v1:172.21.15.203:6789/0 conn(0x7ffbe4143490 legacy=0x7ffbe40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe415dc50 con 0x7ffbe40b1400 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7ffbd8001670 con 0x7ffbe40b1400 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 >> v1:172.21.15.124:6789/0 conn(0x7ffbe40b1400 legacy=0x7ffbe415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbcc005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbcc006130 0x7ffbcc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbcc006f10 0x7ffbcc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe415b520 con 0x7ffbcc006130 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00dc70 con 0x7ffbcc006f10 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc00de30 con 0x7ffbe415e730 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbcc006130 0x7ffbcc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe2ffd640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbcc005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.098+0000 7ffbe37fe640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbcc006f10 0x7ffbcc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 msgr2=0x7ffbcc005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbcc005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbcc006f10 msgr2=0x7ffbcc007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbcc006f10 0x7ffbcc007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3461231020 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc005730 con 0x7ffbcc006130 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe2ffd640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbe415e730 0x7ffbcc005bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbcc006130 0x7ffbcc0069d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c240 tx=0x7ffbcc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cb33c8e63f9dd8 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd8005ad0 con 0x7ffbcc006130 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd8005c30 con 0x7ffbcc006130 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.099+0000 7ffbe1ffb640 1 -- 172.21.15.124:0/3461231020 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd802caf0 con 0x7ffbcc006130 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 -- 172.21.15.124:0/3461231020 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbcc006130 msgr2=0x7ffbcc0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3461231020 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbcc006130 0x7ffbcc0069d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffbd800c240 tx=0x7ffbcc007a00 comp rx=0 tx=0).stop 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe37fe640 1 -- 172.21.15.124:0/3461231020 reap_dead start 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 -- 172.21.15.124:0/3461231020 shutdown_connections 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 -- 172.21.15.124:0/3461231020 >> 172.21.15.124:0/3461231020 conn(0x7ffbe406a4d0 msgr2=0x7ffbe406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 -- 172.21.15.124:0/3461231020 shutdown_connections 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.100+0000 7ffbe9941640 1 -- 172.21.15.124:0/3461231020 wait complete. 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.101+0000 7ffbe9941640 1 Processor -- start 2023-12-13T23:56:43.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.101+0000 7ffbe9941640 1 -- start start 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbcc006130 0x7ffbe4159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40a84d0 0x7ffbe40a8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40af5b0 con 0x7ffbe40aaab0 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40ab450 con 0x7ffbe40a84d0 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe9941640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbe40b03a0 con 0x7ffbcc006130 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37170/0 (socket says 172.21.15.124:37170) 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3226573309 learned_addr learned my addr 172.21.15.124:0/3226573309 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe37fe640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40a84d0 0x7ffbe40a8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe2ffd640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbcc006130 0x7ffbe4159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbcc006130 msgr2=0x7ffbe4159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbcc006130 0x7ffbe4159980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40a84d0 msgr2=0x7ffbe40a8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40a84d0 0x7ffbe40a8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 -- 172.21.15.124:0/3226573309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe40af790 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.102+0000 7ffbe27fc640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c450 tx=0x7ffbd8001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.103+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd803c040 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.103+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbd802cce0 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.103+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbd80072b0 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.103+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbe40afa70 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.103+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbe40b0030 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.104+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbe40b3d90 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.104+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ffbd8032070 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.104+0000 7ffbcbfff640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 0x7ffbd006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.105+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ffbd80ab0d0 con 0x7ffbe40aaab0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.105+0000 7ffbe2ffd640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 0x7ffbd006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:43.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.105+0000 7ffbe2ffd640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 0x7ffbd006d680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40054d0 tx=0x7ffbd4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:43.113 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.109+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ffbd8075c70 con 0x7ffbe40aaab0 2023-12-13T23:56:43.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.287+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffbe4069400 con 0x7ffbe40aaab0 2023-12-13T23:56:43.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.287+0000 7ffbcbfff640 1 -- 172.21.15.124:0/3226573309 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffbd8046080 con 0x7ffbe40aaab0 2023-12-13T23:56:43.286 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:43.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 msgr2=0x7ffbd006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 0x7ffbd006d680 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffbd40054d0 tx=0x7ffbd4009040 comp rx=0 tx=0).stop 2023-12-13T23:56:43.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 msgr2=0x7ffbe40aaf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:43.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7ffbd802c450 tx=0x7ffbd8001a00 comp rx=0 tx=0).stop 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 shutdown_connections 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ffbd006b1c0 0x7ffbd006d680 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ffbe40a84d0 0x7ffbe40a8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ffbe40aaab0 0x7ffbe40aaf10 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 --2- 172.21.15.124:0/3226573309 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ffbcc006130 0x7ffbe4159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 >> 172.21.15.124:0/3226573309 conn(0x7ffbe406a4d0 msgr2=0x7ffbe4071aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 shutdown_connections 2023-12-13T23:56:43.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:43.289+0000 7ffbe9941640 1 -- 172.21.15.124:0/3226573309 wait complete. 2023-12-13T23:56:43.299 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:56:43.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:43.299 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:43.325 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[104701]: fuse finished with error 0 and tester_r 0 2023-12-13T23:56:43.326 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:56:49.428 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:56:49.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:49.428 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:56:49.447 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2023-12-13T23:56:49.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 Processor -- start 2023-12-13T23:56:49.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 -- start start 2023-12-13T23:56:49.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30400b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f30400b43a0 0x7f30400cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30400d00f0 con 0x7f30400d3140 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30400cff00 con 0x7f30400b43a0 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.610+0000 7f3046fd5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30400d1420 con 0x7f304015d420 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f3045572640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f30400d3140 0x7f30400ce320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42376/0 (socket says 172.21.15.124:42376) 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f3045572640 1 -- 172.21.15.124:0/1301797747 learned_addr learned my addr 172.21.15.124:0/1301797747 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303ffff640 1 --2- 172.21.15.124:0/1301797747 >> v2:172.21.15.203:3300/0 conn(0x7f30400b43a0 0x7f30400cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f3044d71640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30400b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503967592 0 0) 0x7f30400d00f0 con 0x7f30400d3140 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028003520 con 0x7f30400d3140 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 321218107 0 0) 0x7f3028003520 con 0x7f30400d3140 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 msgr2=0x7f30400b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30400b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 >> v2:172.21.15.203:3300/0 conn(0x7f30400b43a0 msgr2=0x7f30400cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.613 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 --2- 172.21.15.124:0/1301797747 >> v2:172.21.15.203:3300/0 conn(0x7f30400b43a0 0x7f30400cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30400d03d0 con 0x7f30400d3140 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f30380019f0 con 0x7f30400d3140 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 >> v1:172.21.15.124:6789/0 conn(0x7f30400d3140 legacy=0x7f30400ce320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30280039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3028003f20 0x7f30280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028004d00 0x7f3028005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400d1420 con 0x7f3028003f20 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302800b980 con 0x7f3028004d00 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302800bac0 con 0x7f304015d420 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.611+0000 7f303ffff640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3028003f20 0x7f30280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f3045572640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028004d00 0x7f3028005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f3044d71640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30280039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 -- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 msgr2=0x7f30280039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30280039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 -- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028004d00 msgr2=0x7f3028005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028004d00 0x7f3028005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 -- 172.21.15.124:0/1301797747 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028003520 con 0x7f3028003f20 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f3045572640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028004d00 0x7f3028005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f3044d71640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304015d420 0x7f30280039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.612+0000 7f303ffff640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3028003f20 0x7f30280047c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3034009ac0 tx=0x7f30280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b528d43042878d2 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303402c710 con 0x7f3028003f20 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303402c870 con 0x7f3028003f20 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f303f7fe640 1 -- 172.21.15.124:0/1301797747 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303402cc00 con 0x7f3028003f20 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 -- 172.21.15.124:0/1301797747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3028003f20 msgr2=0x7f30280047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 --2- 172.21.15.124:0/1301797747 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3028003f20 0x7f30280047c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f3034009ac0 tx=0x7f30280057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3045572640 1 -- 172.21.15.124:0/1301797747 reap_dead start 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 -- 172.21.15.124:0/1301797747 shutdown_connections 2023-12-13T23:56:49.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 -- 172.21.15.124:0/1301797747 >> 172.21.15.124:0/1301797747 conn(0x7f304008b000 msgr2=0x7f304009c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:49.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 -- 172.21.15.124:0/1301797747 shutdown_connections 2023-12-13T23:56:49.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.613+0000 7f3046fd5640 1 -- 172.21.15.124:0/1301797747 wait complete. 2023-12-13T23:56:49.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.614+0000 7f3046fd5640 1 Processor -- start 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.614+0000 7f3046fd5640 1 -- start start 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 0x7f3040149a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304014fdd0 0x7f30400ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3040149f70 con 0x7f304015f1d0 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30400ca820 con 0x7f3028003f20 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.615+0000 7f3046fd5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304014cc60 con 0x7f304014fdd0 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37180/0 (socket says 172.21.15.124:37180) 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 -- 172.21.15.124:0/3314625351 learned_addr learned my addr 172.21.15.124:0/3314625351 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3044d71640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 0x7f3040149a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f303ffff640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304014fdd0 0x7f30400ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 -- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304014fdd0 msgr2=0x7f30400ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304014fdd0 0x7f30400ca2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 -- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 msgr2=0x7f3040149a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 0x7f3040149a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 -- 172.21.15.124:0/3314625351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304014d1a0 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3044d71640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 0x7f3040149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3045572640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3038001f30 tx=0x7f303800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f303800eb90 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f303800ecf0 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304014cec0 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038013640 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.616+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30400ccef0 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.618+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3040069400 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.618+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f30380137a0 con 0x7f304015f1d0 2023-12-13T23:56:49.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.618+0000 7f303d7fa640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 0x7f302c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.618+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f3038089f20 con 0x7f304015f1d0 2023-12-13T23:56:49.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.619+0000 7f3044d71640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 0x7f302c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.619+0000 7f3044d71640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 0x7f302c06d6b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3030009a70 tx=0x7f3030008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.623+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3038054c10 con 0x7f304015f1d0 2023-12-13T23:56:49.853 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.854+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) v1 -- 0x7f30400d3fe0 con 0x7f304015f1d0 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.856+0000 7f303d7fa640 1 -- 172.21.15.124:0/3314625351 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v13) v1 ==== 211+0+168 (secure 0 0 0) 0x7f3038058230 con 0x7f304015f1d0 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow rwp" 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:56:49.855 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2023-12-13T23:56:49.856 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 msgr2=0x7f302c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 0x7f302c06d6b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3030009a70 tx=0x7f3030008040 comp rx=0 tx=0).stop 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 msgr2=0x7f30400b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f3038001f30 tx=0x7f303800e530 comp rx=0 tx=0).stop 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 shutdown_connections 2023-12-13T23:56:49.858 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f302c06b1f0 0x7f302c06d6b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f304015f1d0 0x7f30400b4d00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f304014fdd0 0x7f30400ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 --2- 172.21.15.124:0/3314625351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3028003f20 0x7f3040149a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 >> 172.21.15.124:0/3314625351 conn(0x7f304008b000 msgr2=0x7f304006c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 shutdown_connections 2023-12-13T23:56:49.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.858+0000 7f3046fd5640 1 -- 172.21.15.124:0/3314625351 wait complete. 2023-12-13T23:56:49.869 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:56:49.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 Processor -- start 2023-12-13T23:56:49.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 -- start start 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faabc158690 0x7faabc0b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faabc0b0cf0 0x7faabc0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faabc162d80 con 0x7faabc1651b0 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc0b16c0 con 0x7faabc158690 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.970+0000 7faac1e90640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faabc070390 con 0x7faabc0b0cf0 2023-12-13T23:56:49.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faabc158690 0x7faabc0b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faabc158690 0x7faabc0b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37190/0 (socket says 172.21.15.124:37190) 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 -- 172.21.15.124:0/1541998256 learned_addr learned my addr 172.21.15.124:0/1541998256 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabaffd640 1 --2- 172.21.15.124:0/1541998256 >> v2:172.21.15.203:3300/0 conn(0x7faabc0b0cf0 0x7faabc0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 -- 172.21.15.124:0/1541998256 >> v2:172.21.15.203:3300/0 conn(0x7faabc0b0cf0 msgr2=0x7faabc0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 --2- 172.21.15.124:0/1541998256 >> v2:172.21.15.203:3300/0 conn(0x7faabc0b0cf0 0x7faabc0b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 -- 172.21.15.124:0/1541998256 >> v1:172.21.15.203:6789/0 conn(0x7faabc1651b0 legacy=0x7faabc165620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.971+0000 7faabb7fe640 1 -- 172.21.15.124:0/1541998256 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0705a0 con 0x7faabc158690 2023-12-13T23:56:49.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faabb7fe640 1 --2- 172.21.15.124:0/1541998256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc158690 0x7faabc0b44f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faabc15c230 tx=0x7faaac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe5a3d01b382fdd server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faaba7fc640 1 -- 172.21.15.124:0/1541998256 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac031870 con 0x7faabc158690 2023-12-13T23:56:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faaba7fc640 1 -- 172.21.15.124:0/1541998256 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac0319d0 con 0x7faabc158690 2023-12-13T23:56:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faaba7fc640 1 -- 172.21.15.124:0/1541998256 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac031d60 con 0x7faabc158690 2023-12-13T23:56:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 -- 172.21.15.124:0/1541998256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc158690 msgr2=0x7faabc0b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 --2- 172.21.15.124:0/1541998256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc158690 0x7faabc0b44f0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faabc15c230 tx=0x7faaac02b910 comp rx=0 tx=0).stop 2023-12-13T23:56:49.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 -- 172.21.15.124:0/1541998256 shutdown_connections 2023-12-13T23:56:49.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 --2- 172.21.15.124:0/1541998256 >> v2:172.21.15.203:3300/0 conn(0x7faabc0b0cf0 0x7faabc0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 --2- 172.21.15.124:0/1541998256 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc158690 0x7faabc0b44f0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 -- 172.21.15.124:0/1541998256 >> 172.21.15.124:0/1541998256 conn(0x7faabc06a4d0 msgr2=0x7faabc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:49.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 -- 172.21.15.124:0/1541998256 shutdown_connections 2023-12-13T23:56:49.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.972+0000 7faac1e90640 1 -- 172.21.15.124:0/1541998256 wait complete. 2023-12-13T23:56:49.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.973+0000 7faac1e90640 1 Processor -- start 2023-12-13T23:56:49.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 -- start start 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faabc158690 0x7faabc151b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faabc1651b0 0x7faabc152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc1eb8c0 con 0x7faabc0b0cf0 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faabc1eaff0 con 0x7faabc158690 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faac1e90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faabc1fbc50 con 0x7faabc1651b0 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faabb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:37202/0 (socket says 172.21.15.124:37202) 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.974+0000 7faabb7fe640 1 -- 172.21.15.124:0/1583556131 learned_addr learned my addr 172.21.15.124:0/1583556131 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabaffd640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faabc158690 0x7faabc151b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabbfff640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faabc1651b0 0x7faabc152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 -- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faabc1651b0 msgr2=0x7faabc152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faabc1651b0 0x7faabc152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 -- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faabc158690 msgr2=0x7faabc151b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faabc158690 0x7faabc151b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 -- 172.21.15.124:0/1583556131 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac0094d0 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faabb7fe640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7faaac000c00 tx=0x7faaac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac03abe0 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaac03ad40 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac044610 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faabc1eb200 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.975+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faabc13cc50 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.976+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faabc069400 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.977+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7faaac04d450 con 0x7faabc0b0cf0 2023-12-13T23:56:49.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.977+0000 7faaa3fff640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 0x7faaa806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:49.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.977+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7faaac02d040 con 0x7faabc0b0cf0 2023-12-13T23:56:49.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.977+0000 7faabaffd640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 0x7faaa806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:49.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.978+0000 7faabaffd640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 0x7faaa806d680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faabc152b00 tx=0x7faab0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:49.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:49.981+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7faaac0dee90 con 0x7faabc0b0cf0 2023-12-13T23:56:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.185+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7faabc0b3c70 con 0x7faabc0b0cf0 2023-12-13T23:56:50.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.185+0000 7faaa3fff640 1 -- 172.21.15.124:0/1583556131 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+82 (secure 0 0 0) 0x7faaac076da0 con 0x7faabc0b0cf0 2023-12-13T23:56:50.184 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2023-12-13T23:56:50.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 msgr2=0x7faaa806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:50.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 0x7faaa806d680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faabc152b00 tx=0x7faab0008040 comp rx=0 tx=0).stop 2023-12-13T23:56:50.186 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 msgr2=0x7faabc13c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7faaac000c00 tx=0x7faaac0046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 shutdown_connections 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7faaa806b1c0 0x7faaa806d680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faabc158690 0x7faabc151b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faabc1651b0 0x7faabc152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 --2- 172.21.15.124:0/1583556131 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faabc0b0cf0 0x7faabc13c710 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 >> 172.21.15.124:0/1583556131 conn(0x7faabc06a4d0 msgr2=0x7faabc1649d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 shutdown_connections 2023-12-13T23:56:50.187 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:50.187+0000 7faac1e90640 1 -- 172.21.15.124:0/1583556131 wait complete. 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:56:50.197 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:50.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.198 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:56:50.212 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:56:50.213 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42596sec preferred_lft 42596sec 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.214 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:56:50.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.215 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:56:50.278 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:56:50.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.280 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:56:50.335 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:56:50.335 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:56:50.335 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:56:50.336 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:56:50.336 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:56:50.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:56:50.336 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:56:50.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.431 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:56:50.445 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:56:50.446 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:56:50.446 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:50.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.448 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:56:50.513 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:56:50.513 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:50.513 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:50.568 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:56:50.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:56:50.568 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:50.622 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:50.690 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:50.747 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:50.748 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:50.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.750 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:50.801 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:56:50.802 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:56:50.844 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:56:50.872 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:56:50.927 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:56:50.928 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:56:50.929 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:56:50.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:50.930 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:56:50.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:56:50.930+0000 7fd4ccc4c480 -1 init, newargv = 0x55fba15a88b0 newargc=15 2023-12-13T23:56:50.931 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[105784]: starting ceph client 2023-12-13T23:56:50.933 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[105784]: starting fuse 2023-12-13T23:56:50.945 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:56:50.945 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:56:50.945 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:56:50.945 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:56:50.945 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:56:50.945 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:56:50.946 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:56:50.947 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:56:51.068 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.105784.asok 2023-12-13T23:56:51.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:56:51.078 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.105784.asok 2023-12-13T23:56:51.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:51.078 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.105784.asok status 2023-12-13T23:56:51.213 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:56:51.213 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "105784", 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5661, 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:56:51.214 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5661 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1746311428 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1746311428 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:56:51.215 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5661 192.168.0.1:0/1746311428", 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/1746311428", 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 15, 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 38, 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:56:51.216 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:56:51.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:51.222 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:56:51.222 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:51.237 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:56:51.237 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:51.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:51.238 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:51.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:51.305 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2023-12-13T23:56:51.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:51.362 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2023-12-13T23:56:51.420 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:51.420 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:51.476 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:51.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 Processor -- start 2023-12-13T23:56:51.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 -- start start 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02b015e730 0x7f02b00b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02b015e2a0 con 0x7f02b015a4e0 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02b015b520 con 0x7f02b015e730 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.615+0000 7f02b67de640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b0143870 con 0x7f02b0143490 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02b015e730 0x7f02b00b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f02b015e730 0x7f02b00b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48290/0 (socket says 172.21.15.124:48290) 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02b4d7b640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f02b015a4e0 0x7f02b015a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50790/0 (socket says 172.21.15.124:50790) 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 -- 172.21.15.124:0/4023726686 learned_addr learned my addr 172.21.15.124:0/4023726686 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02affff640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 -- 172.21.15.124:0/4023726686 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 msgr2=0x7f02b00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 -- 172.21.15.124:0/4023726686 >> v1:172.21.15.124:6789/0 conn(0x7f02b015a4e0 legacy=0x7f02b015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02af7fe640 1 -- 172.21.15.124:0/4023726686 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b015b240 con 0x7f02b015e730 2023-12-13T23:56:51.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.616+0000 7f02aeffd640 1 -- 172.21.15.124:0/4023726686 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2907939619 0 0) 0x7f02b015e2a0 con 0x7f02b015a4e0 2023-12-13T23:56:51.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02af7fe640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015e730 0x7f02b00b1230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca6a31eeda431c34 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02aeffd640 1 -- 172.21.15.124:0/4023726686 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4030840 con 0x7f02b015e730 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02aeffd640 1 -- 172.21.15.124:0/4023726686 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a40309a0 con 0x7f02b015e730 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02aeffd640 1 -- 172.21.15.124:0/4023726686 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4030d30 con 0x7f02b015e730 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 -- 172.21.15.124:0/4023726686 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015e730 msgr2=0x7f02b00b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015e730 0x7f02b00b1230 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f02a4009870 tx=0x7f02a402b980 comp rx=0 tx=0).stop 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 -- 172.21.15.124:0/4023726686 shutdown_connections 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015e730 0x7f02b00b1230 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 --2- 172.21.15.124:0/4023726686 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 -- 172.21.15.124:0/4023726686 >> 172.21.15.124:0/4023726686 conn(0x7f02b006a4d0 msgr2=0x7f02b006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 -- 172.21.15.124:0/4023726686 shutdown_connections 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.617+0000 7f02b67de640 1 -- 172.21.15.124:0/4023726686 wait complete. 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.618+0000 7f02b67de640 1 Processor -- start 2023-12-13T23:56:51.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 -- start start 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02b015e730 0x7f02b0070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00add20 con 0x7f02b015a4e0 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00ace20 con 0x7f02b015e730 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02b67de640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02b00ad0a0 con 0x7f02b0143490 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.619+0000 7f02af7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48304/0 (socket says 172.21.15.124:48304) 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 -- 172.21.15.124:0/4101904530 learned_addr learned my addr 172.21.15.124:0/4101904530 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02affff640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02b4d7b640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02b015e730 0x7f02b0070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 -- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 msgr2=0x7f02b00b0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 -- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02b015e730 msgr2=0x7f02b0070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02b015e730 0x7f02b0070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 -- 172.21.15.124:0/4101904530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a40094d0 con 0x7f02b015a4e0 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02af7fe640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000c00 tx=0x7f02a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4039a90 con 0x7f02b015a4e0 2023-12-13T23:56:51.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4039bf0 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02b00ad380 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4043410 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.620+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02b00ad940 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.621+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02b00ade60 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.622+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f02a404c3b0 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.622+0000 7f02acff9640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 0x7f029c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.622+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f02a40ac090 con 0x7f02b015a4e0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.622+0000 7f02affff640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 0x7f029c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.623+0000 7f02affff640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 0x7f029c06d680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006fd0 tx=0x7f02a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.625 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.626+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f02b00ade60 con 0x7f02b015a4e0 2023-12-13T23:56:51.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.792+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02b00ade60 con 0x7f02b015a4e0 2023-12-13T23:56:51.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.793+0000 7f02acff9640 1 -- 172.21.15.124:0/4101904530 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f02b00ade60 con 0x7f02b015a4e0 2023-12-13T23:56:51.792 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:51.793 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:51.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 msgr2=0x7f029c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 0x7f029c06d680 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f02a0006fd0 tx=0x7f02a0008040 comp rx=0 tx=0).stop 2023-12-13T23:56:51.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 msgr2=0x7f02b00ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f02a4000c00 tx=0x7f02a4004730 comp rx=0 tx=0).stop 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 shutdown_connections 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f029c06b1c0 0x7f029c06d680 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02b015e730 0x7f02b0070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02b015a4e0 0x7f02b00ac8e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 --2- 172.21.15.124:0/4101904530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02b0143490 0x7f02b00b0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 >> 172.21.15.124:0/4101904530 conn(0x7f02b006a4d0 msgr2=0x7f02b015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 shutdown_connections 2023-12-13T23:56:51.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.795+0000 7f02b67de640 1 -- 172.21.15.124:0/4101904530 wait complete. 2023-12-13T23:56:51.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:51.806 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:51.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.909+0000 7eff41ada640 1 Processor -- start 2023-12-13T23:56:51.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.909+0000 7eff41ada640 1 -- start start 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff41ada640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff3c0d2910 0x7eff3c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff41ada640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3c0b2b10 con 0x7eff3c0d3730 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff41ada640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3c0d1590 con 0x7eff3c15d330 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff41ada640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3c0d3bc0 con 0x7eff3c0d2910 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff3bfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7eff3c0d3730 0x7eff3c0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50800/0 (socket says 172.21.15.124:50800) 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.910+0000 7eff3bfff640 1 -- 172.21.15.124:0/609713335 learned_addr learned my addr 172.21.15.124:0/609713335 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349421929 0 0) 0x7eff3c0b2b10 con 0x7eff3c0d3730 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28003520 con 0x7eff3c0d3730 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff32ffd640 1 --2- 172.21.15.124:0/609713335 >> v2:172.21.15.203:3300/0 conn(0x7eff3c0d2910 0x7eff3c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 455516109 0 0) 0x7eff28003520 con 0x7eff3c0d3730 2023-12-13T23:56:51.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 >> v2:172.21.15.203:3300/0 conn(0x7eff3c0d2910 msgr2=0x7eff3c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 --2- 172.21.15.124:0/609713335 >> v2:172.21.15.203:3300/0 conn(0x7eff3c0d2910 0x7eff3c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 >> v1:172.21.15.203:6789/0 conn(0x7eff3c15d330 legacy=0x7eff3c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c0d1870 con 0x7eff3c0d3730 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7eff340019f0 con 0x7eff3c0d3730 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 >> v1:172.21.15.124:6789/0 conn(0x7eff3c0d3730 legacy=0x7eff3c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 0x7eff28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff280047c0 0x7eff28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28005180 0x7eff2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c0d3bc0 con 0x7eff28003970 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800c120 con 0x7eff280047c0 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800c2e0 con 0x7eff28005180 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 0x7eff28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff32ffd640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff280047c0 0x7eff28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3bfff640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28005180 0x7eff2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 -- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28005180 msgr2=0x7eff2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28005180 0x7eff2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 -- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff280047c0 msgr2=0x7eff28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff280047c0 0x7eff28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3bfff640 1 -- 172.21.15.124:0/609713335 reap_dead start 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff3b7fe640 1 -- 172.21.15.124:0/609713335 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28003520 con 0x7eff28003970 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.911+0000 7eff32ffd640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff280047c0 0x7eff28004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.912+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 0x7eff28003d90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001750 tx=0x7eff280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de96796d57af974 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.912+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0052d0 con 0x7eff28003970 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.912+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c02e050 con 0x7eff28003970 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.912+0000 7eff3affd640 1 -- 172.21.15.124:0/609713335 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c003830 con 0x7eff28003970 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 -- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 msgr2=0x7eff28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 0x7eff28003d90 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7eff2c001750 tx=0x7eff280057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 -- 172.21.15.124:0/609713335 shutdown_connections 2023-12-13T23:56:51.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 --2- 172.21.15.124:0/609713335 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff28003970 0x7eff28003d90 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 -- 172.21.15.124:0/609713335 >> 172.21.15.124:0/609713335 conn(0x7eff3c072020 msgr2=0x7eff3c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:51.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 -- 172.21.15.124:0/609713335 shutdown_connections 2023-12-13T23:56:51.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.913+0000 7eff41ada640 1 -- 172.21.15.124:0/609713335 wait complete. 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.914+0000 7eff41ada640 1 Processor -- start 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.914+0000 7eff41ada640 1 -- start start 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28003970 0x7eff3c14a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 0x7eff3c0c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c156d60 con 0x7eff3c0c87b0 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c158e10 con 0x7eff3c0c8330 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff41ada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c158570 con 0x7eff28003970 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48324/0 (socket says 172.21.15.124:48324) 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 -- 172.21.15.124:0/249997255 learned_addr learned my addr 172.21.15.124:0/249997255 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff32ffd640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 0x7eff3c0c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28003970 0x7eff3c14a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 -- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28003970 msgr2=0x7eff3c14a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28003970 0x7eff3c14a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 -- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 msgr2=0x7eff3c0c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 0x7eff3c0c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 -- 172.21.15.124:0/249997255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c158850 con 0x7eff3c0c87b0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff32ffd640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 0x7eff3c0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.915+0000 7eff3bfff640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0cebd0 tx=0x7eff34007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3400eb00 con 0x7eff3c0c87b0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3400ec80 con 0x7eff3c0c87b0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c158b30 con 0x7eff3c0c87b0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34013420 con 0x7eff3c0c87b0 2023-12-13T23:56:51.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c158200 con 0x7eff3c0c87b0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.916+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c0cfed0 con 0x7eff3c0c87b0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.917+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7eff340135a0 con 0x7eff3c0c87b0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.917+0000 7eff38ff9640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 0x7eff1c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.917+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7eff34089fa0 con 0x7eff3c0c87b0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.917+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 0x7eff1c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.918+0000 7eff3b7fe640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 0x7eff1c06d680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff2c033000 tx=0x7eff2c009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:51.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:51.921+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7eff340bce90 con 0x7eff3c0c87b0 2023-12-13T23:56:52.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.087+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff3c069400 con 0x7eff3c0c87b0 2023-12-13T23:56:52.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.088+0000 7eff38ff9640 1 -- 172.21.15.124:0/249997255 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff34054bd0 con 0x7eff3c0c87b0 2023-12-13T23:56:52.087 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 msgr2=0x7eff1c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 0x7eff1c06d680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7eff2c033000 tx=0x7eff2c009030 comp rx=0 tx=0).stop 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 msgr2=0x7eff3c1f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0cebd0 tx=0x7eff34007bc0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 shutdown_connections 2023-12-13T23:56:52.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff1c06b1c0 0x7eff1c06d680 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff3c0c8330 0x7eff3c0c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff3c0c87b0 0x7eff3c1f66e0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 --2- 172.21.15.124:0/249997255 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff28003970 0x7eff3c14a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 >> 172.21.15.124:0/249997255 conn(0x7eff3c072020 msgr2=0x7eff3c0cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 shutdown_connections 2023-12-13T23:56:52.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.090+0000 7eff41ada640 1 -- 172.21.15.124:0/249997255 wait complete. 2023-12-13T23:56:52.100 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:52.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.203+0000 7fa38f1a9640 1 Processor -- start 2023-12-13T23:56:52.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.203+0000 7fa38f1a9640 1 -- start start 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38f1a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa388143490 0x7fa3880b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38f1a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa38815abf0 0x7fa3880b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38f1a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa38815e2a0 con 0x7fa38815e730 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38f1a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa38815b520 con 0x7fa388143490 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38f1a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa388143870 con 0x7fa38815abf0 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38cf45640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa388143490 0x7fa3880b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38d746640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa38815abf0 0x7fa3880b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa387fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa38815e730 0x7fa38815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:41466/0 (socket says 172.21.15.124:41466) 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa387fff640 1 -- 172.21.15.124:0/1951952105 learned_addr learned my addr 172.21.15.124:0/1951952105 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38cf45640 1 -- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa38815abf0 msgr2=0x7fa3880b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38cf45640 1 --2- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa38815abf0 0x7fa3880b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38cf45640 1 -- 172.21.15.124:0/1951952105 >> v1:172.21.15.203:6789/0 conn(0x7fa38815e730 legacy=0x7fa38815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.204+0000 7fa38cf45640 1 -- 172.21.15.124:0/1951952105 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38815b240 con 0x7fa388143490 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.205+0000 7fa38cf45640 1 --2- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa388143490 0x7fa3880b0cf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097a0 tx=0x7fa37c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22289c5f27c276cc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.205+0000 7fa3877fe640 1 -- 172.21.15.124:0/1951952105 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c02d040 con 0x7fa388143490 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.205+0000 7fa3877fe640 1 -- 172.21.15.124:0/1951952105 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa37c031810 con 0x7fa388143490 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.205+0000 7fa3877fe640 1 -- 172.21.15.124:0/1951952105 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa37c031ba0 con 0x7fa388143490 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 -- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa388143490 msgr2=0x7fa3880b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa388143490 0x7fa3880b0cf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa37c0097a0 tx=0x7fa37c02b8b0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 -- 172.21.15.124:0/1951952105 shutdown_connections 2023-12-13T23:56:52.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa38815abf0 0x7fa3880b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/1951952105 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa388143490 0x7fa3880b0cf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 -- 172.21.15.124:0/1951952105 >> 172.21.15.124:0/1951952105 conn(0x7fa38806a4d0 msgr2=0x7fa38806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 -- 172.21.15.124:0/1951952105 shutdown_connections 2023-12-13T23:56:52.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.206+0000 7fa38f1a9640 1 -- 172.21.15.124:0/1951952105 wait complete. 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.207+0000 7fa38f1a9640 1 Processor -- start 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.207+0000 7fa38f1a9640 1 -- start start 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa388143490 0x7fa388159980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa38815e730 0x7fa3881f0f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880aa030 con 0x7fa38815abf0 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880a93b0 con 0x7fa38815e730 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38f1a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3880a9690 con 0x7fa388143490 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48330/0 (socket says 172.21.15.124:48330) 2023-12-13T23:56:52.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 -- 172.21.15.124:0/4023779832 learned_addr learned my addr 172.21.15.124:0/4023779832 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38d746640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa38815e730 0x7fa3881f0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa38cf45640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa388143490 0x7fa388159980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 -- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa388143490 msgr2=0x7fa388159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa388143490 0x7fa388159980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 -- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa38815e730 msgr2=0x7fa3881f0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa38815e730 0x7fa3881f0f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 -- 172.21.15.124:0/4023779832 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa37c009400 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.208+0000 7fa387fff640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa37400c480 tx=0x7fa37400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.209+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa374005ce0 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.209+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa374005e40 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.209+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3740072f0 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.209+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3880a9970 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.209+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa388159ec0 con 0x7fa38815abf0 2023-12-13T23:56:52.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.210+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3880aa170 con 0x7fa38815abf0 2023-12-13T23:56:52.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.210+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa374007510 con 0x7fa38815abf0 2023-12-13T23:56:52.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.210+0000 7fa3857fa640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 0x7fa36c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.211+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fa374088fa0 con 0x7fa38815abf0 2023-12-13T23:56:52.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.211+0000 7fa38cf45640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 0x7fa36c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.211+0000 7fa38cf45640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 0x7fa36c06d680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa37c036000 tx=0x7fa37c03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.215+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa3740bbe90 con 0x7fa38815abf0 2023-12-13T23:56:52.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.383+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa388069400 con 0x7fa38815abf0 2023-12-13T23:56:52.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.383+0000 7fa3857fa640 1 -- 172.21.15.124:0/4023779832 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa3740170a0 con 0x7fa38815abf0 2023-12-13T23:56:52.382 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:52.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 msgr2=0x7fa36c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 0x7fa36c06d680 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa37c036000 tx=0x7fa37c03c040 comp rx=0 tx=0).stop 2023-12-13T23:56:52.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 msgr2=0x7fa3880adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fa37400c480 tx=0x7fa37400c940 comp rx=0 tx=0).stop 2023-12-13T23:56:52.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 shutdown_connections 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa36c06b1c0 0x7fa36c06d680 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa38815e730 0x7fa3881f0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa38815abf0 0x7fa3880adcc0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 --2- 172.21.15.124:0/4023779832 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa388143490 0x7fa388159980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 >> 172.21.15.124:0/4023779832 conn(0x7fa38806a4d0 msgr2=0x7fa3880b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 shutdown_connections 2023-12-13T23:56:52.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.385+0000 7fa38f1a9640 1 -- 172.21.15.124:0/4023779832 wait complete. 2023-12-13T23:56:52.395 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:52.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.499+0000 7fa85206b640 1 Processor -- start 2023-12-13T23:56:52.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.499+0000 7fa85206b640 1 -- start start 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa85206b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa84c0ceb70 0x7fa84c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa85206b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fa84c0b4370 0x7fa84c0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa85206b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c071770 con 0x7fa84c144980 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa85206b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c15ced0 con 0x7fa84c0ceb70 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa85206b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0b4ce0 con 0x7fa84c0b4370 2023-12-13T23:56:52.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa84c0ceb70 0x7fa84c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa84c0ceb70 0x7fa84c0cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48344/0 (socket says 172.21.15.124:48344) 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 -- 172.21.15.124:0/1803574889 learned_addr learned my addr 172.21.15.124:0/1803574889 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84bfff640 1 --2- 172.21.15.124:0/1803574889 >> v2:172.21.15.203:3300/0 conn(0x7fa84c0b4370 0x7fa84c0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 -- 172.21.15.124:0/1803574889 >> v2:172.21.15.203:3300/0 conn(0x7fa84c0b4370 msgr2=0x7fa84c0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 --2- 172.21.15.124:0/1803574889 >> v2:172.21.15.203:3300/0 conn(0x7fa84c0b4370 0x7fa84c0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 -- 172.21.15.124:0/1803574889 >> v1:172.21.15.203:6789/0 conn(0x7fa84c144980 legacy=0x7fa84c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.500+0000 7fa84affd640 1 -- 172.21.15.124:0/1803574889 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c0ccdc0 con 0x7fa84c0ceb70 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.501+0000 7fa84affd640 1 --2- 172.21.15.124:0/1803574889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0ceb70 0x7fa84c0cc450 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0716e0 tx=0x7fa84002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7fc8b94ea716894 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.501+0000 7fa84a7fc640 1 -- 172.21.15.124:0/1803574889 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840030860 con 0x7fa84c0ceb70 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.501+0000 7fa84a7fc640 1 -- 172.21.15.124:0/1803574889 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400309c0 con 0x7fa84c0ceb70 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.501+0000 7fa84a7fc640 1 -- 172.21.15.124:0/1803574889 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa840030d50 con 0x7fa84c0ceb70 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 -- 172.21.15.124:0/1803574889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0ceb70 msgr2=0x7fa84c0cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 --2- 172.21.15.124:0/1803574889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0ceb70 0x7fa84c0cc450 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0716e0 tx=0x7fa84002b9a0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 -- 172.21.15.124:0/1803574889 shutdown_connections 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 --2- 172.21.15.124:0/1803574889 >> v2:172.21.15.203:3300/0 conn(0x7fa84c0b4370 0x7fa84c0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 --2- 172.21.15.124:0/1803574889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0ceb70 0x7fa84c0cc450 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 -- 172.21.15.124:0/1803574889 >> 172.21.15.124:0/1803574889 conn(0x7fa84c06a4d0 msgr2=0x7fa84c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 -- 172.21.15.124:0/1803574889 shutdown_connections 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.502+0000 7fa85206b640 1 -- 172.21.15.124:0/1803574889 wait complete. 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.503+0000 7fa85206b640 1 Processor -- start 2023-12-13T23:56:52.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.503+0000 7fa85206b640 1 -- start start 2023-12-13T23:56:52.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 0x7fa84c1f0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 0x7fa84c1f1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c1f25e0 con 0x7fa84c0b4370 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c1f1960 con 0x7fa84c144980 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c1f1c40 con 0x7fa84c0ceb70 2023-12-13T23:56:52.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48356/0 (socket says 172.21.15.124:48356) 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 -- 172.21.15.124:0/1228541805 learned_addr learned my addr 172.21.15.124:0/1228541805 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84affd640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 0x7fa84c1f0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84bfff640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 0x7fa84c1f1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 -- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 msgr2=0x7fa84c1f0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 0x7fa84c1f0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 -- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 msgr2=0x7fa84c1f1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 0x7fa84c1f1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 -- 172.21.15.124:0/1228541805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8400094d0 con 0x7fa84c0b4370 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84affd640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 0x7fa84c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84bfff640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 0x7fa84c1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:52.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa84b7fe640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00c400 tx=0x7fa83c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c0052d0 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c005430 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c1f1f20 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.504+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c003830 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.505+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c140b50 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.505+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c1f2720 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.506+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fa83c003990 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.506+0000 7fa833fff640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 0x7fa83806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.506+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fa83c089110 con 0x7fa84c0b4370 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.507+0000 7fa84affd640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 0x7fa83806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.507+0000 7fa84affd640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 0x7fa83806d680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa840036000 tx=0x7fa84003b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.511+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fa84c1f2720 con 0x7fa84c0b4370 2023-12-13T23:56:52.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.676+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa84c0b1e30 con 0x7fa84c0b4370 2023-12-13T23:56:52.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.676+0000 7fa833fff640 1 -- 172.21.15.124:0/1228541805 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa83c053cb0 con 0x7fa84c0b4370 2023-12-13T23:56:52.675 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:52.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 msgr2=0x7fa83806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 0x7fa83806d680 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa840036000 tx=0x7fa84003b040 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 msgr2=0x7fa84c0b0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa83c00c400 tx=0x7fa83c00c8c0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 shutdown_connections 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fa83806b1c0 0x7fa83806d680 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fa84c144980 0x7fa84c1f1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fa84c0ceb70 0x7fa84c1f0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 --2- 172.21.15.124:0/1228541805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa84c0b4370 0x7fa84c0b0ba0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 >> 172.21.15.124:0/1228541805 conn(0x7fa84c06a4d0 msgr2=0x7fa84c1590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 shutdown_connections 2023-12-13T23:56:52.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.678+0000 7fa85206b640 1 -- 172.21.15.124:0/1228541805 wait complete. 2023-12-13T23:56:52.688 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:52.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.792+0000 7f27515a9640 1 Processor -- start 2023-12-13T23:56:52.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.792+0000 7f27515a9640 1 -- start start 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f27515a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 0x7f274c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f27515a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f274c150940 0x7f274c0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f27515a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f274c0b49f0 con 0x7f274c0b3390 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f27515a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f274c0b3de0 con 0x7f274c150940 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f27515a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0b40c0 con 0x7f274c150560 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f274a7fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f274c0b3390 0x7f274c15d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50812/0 (socket says 172.21.15.124:50812) 2023-12-13T23:56:52.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.793+0000 7f274a7fc640 1 -- 172.21.15.124:0/2184447330 learned_addr learned my addr 172.21.15.124:0/2184447330 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274affd640 1 --2- 172.21.15.124:0/2184447330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 0x7f274c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274b7fe640 1 --2- 172.21.15.124:0/2184447330 >> v2:172.21.15.203:3300/0 conn(0x7f274c150940 0x7f274c0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f2749ffb640 1 -- 172.21.15.124:0/2184447330 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351849042 0 0) 0x7f274c0b49f0 con 0x7f274c0b3390 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f2749ffb640 1 -- 172.21.15.124:0/2184447330 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2734003520 con 0x7f274c0b3390 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274affd640 1 -- 172.21.15.124:0/2184447330 >> v2:172.21.15.203:3300/0 conn(0x7f274c150940 msgr2=0x7f274c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274affd640 1 --2- 172.21.15.124:0/2184447330 >> v2:172.21.15.203:3300/0 conn(0x7f274c150940 0x7f274c0ce0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274affd640 1 -- 172.21.15.124:0/2184447330 >> v1:172.21.15.124:6789/0 conn(0x7f274c0b3390 legacy=0x7f274c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.794+0000 7f274affd640 1 -- 172.21.15.124:0/2184447330 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f274c0b43a0 con 0x7f274c150560 2023-12-13T23:56:52.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f274affd640 1 --2- 172.21.15.124:0/2184447330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 0x7f274c0b2e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f273c009d90 tx=0x7f273c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56a037bd86d774bb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f2749ffb640 1 -- 172.21.15.124:0/2184447330 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c02cc50 con 0x7f274c150560 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f2749ffb640 1 -- 172.21.15.124:0/2184447330 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273c02cdb0 con 0x7f274c150560 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f2749ffb640 1 -- 172.21.15.124:0/2184447330 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273c034640 con 0x7f274c150560 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 -- 172.21.15.124:0/2184447330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 msgr2=0x7f274c0b2e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 --2- 172.21.15.124:0/2184447330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 0x7f274c0b2e50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f273c009d90 tx=0x7f273c02bea0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 -- 172.21.15.124:0/2184447330 shutdown_connections 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 --2- 172.21.15.124:0/2184447330 >> v2:172.21.15.203:3300/0 conn(0x7f274c150940 0x7f274c0ce0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 --2- 172.21.15.124:0/2184447330 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c150560 0x7f274c0b2e50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 -- 172.21.15.124:0/2184447330 >> 172.21.15.124:0/2184447330 conn(0x7f274c06a4d0 msgr2=0x7f274c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 -- 172.21.15.124:0/2184447330 shutdown_connections 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.795+0000 7f27515a9640 1 -- 172.21.15.124:0/2184447330 wait complete. 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 Processor -- start 2023-12-13T23:56:52.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 -- start start 2023-12-13T23:56:52.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 0x7f274c14ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 0x7f274c140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c152380 con 0x7f274c0af270 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0afcc0 con 0x7f274c150560 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f27515a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f274c0af700 con 0x7f274c0b3390 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.797+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48358/0 (socket says 172.21.15.124:48358) 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 -- 172.21.15.124:0/721370054 learned_addr learned my addr 172.21.15.124:0/721370054 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274a7fc640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 0x7f274c140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274affd640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 0x7f274c14ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 -- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 msgr2=0x7f274c14ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 0x7f274c14ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 -- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 msgr2=0x7f274c140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 0x7f274c140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 -- 172.21.15.124:0/721370054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273c009a70 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274a7fc640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 0x7f274c140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274affd640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 0x7f274c14ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f274b7fe640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f274400e820 tx=0x7f274400ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2744009510 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f274c0af9e0 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f274400ee70 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f274400cb80 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.798+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f274c152870 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.799+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f274c152a90 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.799+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f274400cce0 con 0x7f274c0af270 2023-12-13T23:56:52.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.800+0000 7f27337fe640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 0x7f273806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:52.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.800+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f2744089ca0 con 0x7f274c0af270 2023-12-13T23:56:52.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.800+0000 7f274affd640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 0x7f273806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:52.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.801+0000 7f274affd640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 0x7f273806d6b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f273c036000 tx=0x7f273c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:52.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.804+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f27440547c0 con 0x7f274c0af270 2023-12-13T23:56:52.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.972+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f274c15de90 con 0x7f274c0af270 2023-12-13T23:56:52.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.972+0000 7f27337fe640 1 -- 172.21.15.124:0/721370054 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f2744057de0 con 0x7f274c0af270 2023-12-13T23:56:52.971 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 msgr2=0x7f273806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 0x7f273806d6b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f273c036000 tx=0x7f273c030000 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 msgr2=0x7f274c138ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f274400e820 tx=0x7f274400ec20 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 shutdown_connections 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f273806b1f0 0x7f273806d6b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f274c150560 0x7f274c140af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f274c0af270 0x7f274c138ac0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 --2- 172.21.15.124:0/721370054 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f274c0b3390 0x7f274c14ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 >> 172.21.15.124:0/721370054 conn(0x7f274c06a4d0 msgr2=0x7f274c071040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 shutdown_connections 2023-12-13T23:56:52.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:52.974+0000 7f27515a9640 1 -- 172.21.15.124:0/721370054 wait complete. 2023-12-13T23:56:52.984 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:53.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.085+0000 7f9183d2c640 1 Processor -- start 2023-12-13T23:56:53.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.085+0000 7f9183d2c640 1 -- start start 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f9183d2c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f917c15d330 0x7f917c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f9183d2c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f917c0d2910 0x7f917c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f9183d2c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f917c0b2b10 con 0x7f917c0d3730 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f9183d2c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917c0d3bc0 con 0x7f917c0d2910 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f9183d2c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f917c0d1590 con 0x7f917c15d330 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f91812c7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f917c0d2910 0x7f917c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f91812c7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f917c0d2910 0x7f917c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48360/0 (socket says 172.21.15.124:48360) 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.086+0000 7f91812c7640 1 -- 172.21.15.124:0/3813660086 learned_addr learned my addr 172.21.15.124:0/3813660086 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f9181ac8640 1 --2- 172.21.15.124:0/3813660086 >> v2:172.21.15.203:3300/0 conn(0x7f917c15d330 0x7f917c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f91812c7640 1 -- 172.21.15.124:0/3813660086 >> v2:172.21.15.203:3300/0 conn(0x7f917c15d330 msgr2=0x7f917c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f91812c7640 1 --2- 172.21.15.124:0/3813660086 >> v2:172.21.15.203:3300/0 conn(0x7f917c15d330 0x7f917c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f91812c7640 1 -- 172.21.15.124:0/3813660086 >> v1:172.21.15.203:6789/0 conn(0x7f917c0d3730 legacy=0x7f917c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f91812c7640 1 -- 172.21.15.124:0/3813660086 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f917c0d1870 con 0x7f917c0d2910 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f91812c7640 1 --2- 172.21.15.124:0/3813660086 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c0d31f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f917c0d0d10 tx=0x7f916c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bae2e966232421d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.087+0000 7f9180ac6640 1 -- 172.21.15.124:0/3813660086 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c0317e0 con 0x7f917c0d2910 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9180ac6640 1 -- 172.21.15.124:0/3813660086 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f916c031940 con 0x7f917c0d2910 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9180ac6640 1 -- 172.21.15.124:0/3813660086 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f916c031cd0 con 0x7f917c0d2910 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 -- 172.21.15.124:0/3813660086 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 msgr2=0x7f917c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 --2- 172.21.15.124:0/3813660086 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c0d31f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f917c0d0d10 tx=0x7f916c02b880 comp rx=0 tx=0).stop 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 -- 172.21.15.124:0/3813660086 shutdown_connections 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 --2- 172.21.15.124:0/3813660086 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c0d31f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 --2- 172.21.15.124:0/3813660086 >> v2:172.21.15.203:3300/0 conn(0x7f917c15d330 0x7f917c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 -- 172.21.15.124:0/3813660086 >> 172.21.15.124:0/3813660086 conn(0x7f917c072020 msgr2=0x7f917c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 -- 172.21.15.124:0/3813660086 shutdown_connections 2023-12-13T23:56:53.089 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.088+0000 7f9183d2c640 1 -- 172.21.15.124:0/3813660086 wait complete. 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.089+0000 7f9183d2c640 1 Processor -- start 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.089+0000 7f9183d2c640 1 -- start start 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 0x7f917c0c9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 0x7f917c1f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c0c9bc0 con 0x7f917c0d2910 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c1600f0 con 0x7f917c15d330 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9183d2c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c15f850 con 0x7f917c0d3730 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9181ac8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9181ac8640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48376/0 (socket says 172.21.15.124:48376) 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9181ac8640 1 -- 172.21.15.124:0/1482382834 learned_addr learned my addr 172.21.15.124:0/1482382834 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f91822c9640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 0x7f917c1f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f91812c7640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 0x7f917c0c9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.090+0000 7f9181ac8640 1 -- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 msgr2=0x7f917c0c9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9181ac8640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 0x7f917c0c9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9181ac8640 1 -- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 msgr2=0x7f917c1f6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9181ac8640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 0x7f917c1f6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9181ac8640 1 -- 172.21.15.124:0/1482382834 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c0094d0 con 0x7f917c0d2910 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f91822c9640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 0x7f917c1f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:53.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f91812c7640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 0x7f917c0c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9181ac8640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9178007e70 tx=0x7f917800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178014620 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9178014780 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917c15fb30 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9178014b30 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.091+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c14a270 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.092+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f917c14a430 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.092+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9178014c90 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.093+0000 7f91727fc640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 0x7f916006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.093+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f917808a440 con 0x7f917c0d2910 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.093+0000 7f91812c7640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 0x7f916006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.093+0000 7f91812c7640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 0x7f916006d680 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f916c036000 tx=0x7f916c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.097+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9178055070 con 0x7f917c0d2910 2023-12-13T23:56:53.273 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.274+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f917c069400 con 0x7f917c0d2910 2023-12-13T23:56:53.273 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.274+0000 7f91727fc640 1 -- 172.21.15.124:0/1482382834 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9178058690 con 0x7f917c0d2910 2023-12-13T23:56:53.274 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:53.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 msgr2=0x7f916006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 0x7f916006d680 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f916c036000 tx=0x7f916c03d040 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 msgr2=0x7f917c15f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f9178007e70 tx=0x7f917800e4b0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 shutdown_connections 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f916006b1c0 0x7f916006d680 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f917c15d330 0x7f917c1f6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f917c0d3730 0x7f917c0c9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 --2- 172.21.15.124:0/1482382834 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f917c0d2910 0x7f917c15f140 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 >> 172.21.15.124:0/1482382834 conn(0x7f917c072020 msgr2=0x7f917c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 shutdown_connections 2023-12-13T23:56:53.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.276+0000 7f9183d2c640 1 -- 172.21.15.124:0/1482382834 wait complete. 2023-12-13T23:56:53.286 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:56:53.286 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:56:53.303 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:56:53.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.443+0000 7f8c7347c640 1 Processor -- start 2023-12-13T23:56:53.442 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.443+0000 7f8c7347c640 1 -- start start 2023-12-13T23:56:53.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c7347c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 0x7f8c6c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c7347c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8c6c0b3390 0x7f8c6c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c7347c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c6c0b49f0 con 0x7f8c6c150940 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c7347c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c0b3de0 con 0x7f8c6c0b3390 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c7347c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c6c0b40c0 con 0x7f8c6c150560 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8c6c0b3390 0x7f8c6c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8c6c0b3390 0x7f8c6c15d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48390/0 (socket says 172.21.15.124:48390) 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 -- 172.21.15.124:0/775842516 learned_addr learned my addr 172.21.15.124:0/775842516 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c71218640 1 --2- 172.21.15.124:0/775842516 >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 0x7f8c6c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 -- 172.21.15.124:0/775842516 >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 msgr2=0x7f8c6c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 --2- 172.21.15.124:0/775842516 >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 0x7f8c6c0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 -- 172.21.15.124:0/775842516 >> v1:172.21.15.203:6789/0 conn(0x7f8c6c150940 legacy=0x7f8c6c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c70a17640 1 -- 172.21.15.124:0/775842516 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6c0b43a0 con 0x7f8c6c0b3390 2023-12-13T23:56:53.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.444+0000 7f8c71218640 1 --2- 172.21.15.124:0/775842516 >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 0x7f8c6c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.445+0000 7f8c70a17640 1 --2- 172.21.15.124:0/775842516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c15d200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c072840 tx=0x7f8c5c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce09382246885a0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.445+0000 7f8c63fff640 1 -- 172.21.15.124:0/775842516 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c031870 con 0x7f8c6c0b3390 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.445+0000 7f8c63fff640 1 -- 172.21.15.124:0/775842516 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c5c0319d0 con 0x7f8c6c0b3390 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.445+0000 7f8c63fff640 1 -- 172.21.15.124:0/775842516 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c5c031d60 con 0x7f8c6c0b3390 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 -- 172.21.15.124:0/775842516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0b3390 msgr2=0x7f8c6c15d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 --2- 172.21.15.124:0/775842516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c15d200 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f8c6c072840 tx=0x7f8c5c02b910 comp rx=0 tx=0).stop 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 -- 172.21.15.124:0/775842516 shutdown_connections 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 --2- 172.21.15.124:0/775842516 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c15d200 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 --2- 172.21.15.124:0/775842516 >> v2:172.21.15.203:3300/0 conn(0x7f8c6c150560 0x7f8c6c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 -- 172.21.15.124:0/775842516 >> 172.21.15.124:0/775842516 conn(0x7f8c6c06a4d0 msgr2=0x7f8c6c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 -- 172.21.15.124:0/775842516 shutdown_connections 2023-12-13T23:56:53.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.446+0000 7f8c7347c640 1 -- 172.21.15.124:0/775842516 wait complete. 2023-12-13T23:56:53.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.447+0000 7f8c7347c640 1 Processor -- start 2023-12-13T23:56:53.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.447+0000 7f8c7347c640 1 -- start start 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c14b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8c6c150560 0x7f8c6c0b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c15a760 con 0x7f8c6c0717b0 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c1579d0 con 0x7f8c6c0b3390 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c7347c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c6c157cb0 con 0x7f8c6c150560 2023-12-13T23:56:53.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48406/0 (socket says 172.21.15.124:48406) 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 -- 172.21.15.124:0/1495315870 learned_addr learned my addr 172.21.15.124:0/1495315870 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c70a17640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8c6c150560 0x7f8c6c0b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71218640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c14b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 -- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8c6c150560 msgr2=0x7f8c6c0b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8c6c150560 0x7f8c6c0b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 -- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6c0b3390 msgr2=0x7f8c6c14b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c14b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 -- 172.21.15.124:0/1495315870 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5c0094d0 con 0x7f8c6c0717b0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.448+0000 7f8c71a19640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b410 tx=0x7f8c6400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.449+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64004020 con 0x7f8c6c0717b0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.449+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c64004180 con 0x7f8c6c0717b0 2023-12-13T23:56:53.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.449+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c6c156c70 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.449+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c64011630 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.449+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c6c14b8d0 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.450+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c6c157190 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.450+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8c64011790 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.450+0000 7f8c61ffb640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 0x7f8c5006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.450+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f8c640898b0 con 0x7f8c6c0717b0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.450+0000 7f8c71218640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 0x7f8c5006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.451+0000 7f8c71218640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 0x7f8c5006d6b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c680075b0 tx=0x7f8c68007540 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.455+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8c6c157190 con 0x7f8c6c0717b0 2023-12-13T23:56:53.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.624+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c6c157190 con 0x7f8c6c0717b0 2023-12-13T23:56:53.624 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.625+0000 7f8c61ffb640 1 -- 172.21.15.124:0/1495315870 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f8c6c157190 con 0x7f8c6c0717b0 2023-12-13T23:56:53.624 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:56:53.624 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:56:53.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 msgr2=0x7f8c5006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 0x7f8c5006d6b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8c680075b0 tx=0x7f8c68007540 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 msgr2=0x7f8c6c1498d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f8c6400b410 tx=0x7f8c6400b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 shutdown_connections 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8c5006b1f0 0x7f8c5006d6b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8c6c0717b0 0x7f8c6c1498d0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8c6c150560 0x7f8c6c0b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 --2- 172.21.15.124:0/1495315870 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8c6c0b3390 0x7f8c6c14b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 >> 172.21.15.124:0/1495315870 conn(0x7f8c6c06a4d0 msgr2=0x7f8c6c14fa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 shutdown_connections 2023-12-13T23:56:53.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.627+0000 7f8c7347c640 1 -- 172.21.15.124:0/1495315870 wait complete. 2023-12-13T23:56:53.637 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:56:53.638 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:56:53.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 Processor -- start 2023-12-13T23:56:53.741 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 -- start start 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 0x7f733c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f733c0b2b10 con 0x7f733c0d2910 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f733c0d1590 con 0x7f733c15d330 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.742+0000 7f7342dc5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0d1870 con 0x7f733c0d3900 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7340b61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 0x7f733c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f733bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f733c0d2910 0x7f733c0d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:41506/0 (socket says 172.21.15.124:41506) 2023-12-13T23:56:53.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:35116/0 (socket says 172.21.15.124:35116) 2023-12-13T23:56:53.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f733bfff640 1 -- 172.21.15.124:0/360126247 learned_addr learned my addr 172.21.15.124:0/360126247 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:53.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f733b7fe640 1 -- 172.21.15.124:0/360126247 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657109700 0 0) 0x7f733c0b2b10 con 0x7f733c0d2910 2023-12-13T23:56:53.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f733b7fe640 1 -- 172.21.15.124:0/360126247 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c003520 con 0x7f733c0d2910 2023-12-13T23:56:53.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 -- 172.21.15.124:0/360126247 >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 msgr2=0x7f733c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:53.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 --2- 172.21.15.124:0/360126247 >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 0x7f733c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 -- 172.21.15.124:0/360126247 >> v1:172.21.15.203:6789/0 conn(0x7f733c0d2910 legacy=0x7f733c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7341362640 1 -- 172.21.15.124:0/360126247 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f733c0d1b50 con 0x7f733c0d3900 2023-12-13T23:56:53.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.743+0000 7f7340b61640 1 --2- 172.21.15.124:0/360126247 >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 0x7f733c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:56:53.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.744+0000 7f7341362640 1 --2- 172.21.15.124:0/360126247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7334009d90 tx=0x7f733402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8b802feeebd1b8f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.744+0000 7f733b7fe640 1 -- 172.21.15.124:0/360126247 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733402cbc0 con 0x7f733c0d3900 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.744+0000 7f733b7fe640 1 -- 172.21.15.124:0/360126247 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733402cd20 con 0x7f733c0d3900 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.744+0000 7f733b7fe640 1 -- 172.21.15.124:0/360126247 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7334034660 con 0x7f733c0d3900 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 -- 172.21.15.124:0/360126247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 msgr2=0x7f733c0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 --2- 172.21.15.124:0/360126247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7334009d90 tx=0x7f733402be10 comp rx=0 tx=0).stop 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 -- 172.21.15.124:0/360126247 shutdown_connections 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 --2- 172.21.15.124:0/360126247 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c0d3900 0x7f733c0b6390 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 --2- 172.21.15.124:0/360126247 >> v2:172.21.15.203:3300/0 conn(0x7f733c15d330 0x7f733c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 -- 172.21.15.124:0/360126247 >> 172.21.15.124:0/360126247 conn(0x7f733c072020 msgr2=0x7f733c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 -- 172.21.15.124:0/360126247 shutdown_connections 2023-12-13T23:56:53.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.745+0000 7f7342dc5640 1 -- 172.21.15.124:0/360126247 wait complete. 2023-12-13T23:56:53.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.746+0000 7f7342dc5640 1 Processor -- start 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.746+0000 7f7342dc5640 1 -- start start 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 0x7f733c1430d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c15d330 0x7f733c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0c9c20 con 0x7f733c0d2910 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0cb520 con 0x7f733c0d3900 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f733c0cb800 con 0x7f733c15d330 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48420/0 (socket says 172.21.15.124:48420) 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 -- 172.21.15.124:0/2579203587 learned_addr learned my addr 172.21.15.124:0/2579203587 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f733bfff640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 0x7f733c1430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7341362640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c15d330 0x7f733c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 -- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c15d330 msgr2=0x7f733c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c15d330 0x7f733c0c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 -- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 msgr2=0x7f733c1430d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 0x7f733c1430d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 -- 172.21.15.124:0/2579203587 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7330009400 con 0x7f733c0d2910 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f733bfff640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 0x7f733c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7340b61640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f733000ebe0 tx=0x7f733000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330009760 con 0x7f733c0d2910 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f733000cc30 con 0x7f733c0d2910 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7334009a70 con 0x7f733c0d2910 2023-12-13T23:56:53.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.747+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f733000eee0 con 0x7f733c0d2910 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.748+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733c0cbdc0 con 0x7f733c0d2910 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.748+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f733c069400 con 0x7f733c0d2910 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.749+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7330013470 con 0x7f733c0d2910 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.749+0000 7f73397fa640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 0x7f732006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.749+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7330089e70 con 0x7f733c0d2910 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.750+0000 7f733bfff640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 0x7f732006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:53.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.750+0000 7f733bfff640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 0x7f732006d680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f733c152ee0 tx=0x7f7328009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:53.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.753+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7330054aa0 con 0x7f733c0d2910 2023-12-13T23:56:53.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.919+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f733c0d0f80 con 0x7f733c0d2910 2023-12-13T23:56:53.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.920+0000 7f73397fa640 1 -- 172.21.15.124:0/2579203587 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73300580c0 con 0x7f733c0d2910 2023-12-13T23:56:53.919 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:56:53.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 msgr2=0x7f732006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 0x7f732006d680 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f733c152ee0 tx=0x7f7328009040 comp rx=0 tx=0).stop 2023-12-13T23:56:53.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 msgr2=0x7f733c144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:53.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f733000ebe0 tx=0x7f733000c520 comp rx=0 tx=0).stop 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 shutdown_connections 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f732006b1c0 0x7f732006d680 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f733c0d3900 0x7f733c1430d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f733c15d330 0x7f733c0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 --2- 172.21.15.124:0/2579203587 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f733c0d2910 0x7f733c144da0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 >> 172.21.15.124:0/2579203587 conn(0x7f733c072020 msgr2=0x7f733c0d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 shutdown_connections 2023-12-13T23:56:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:53.922+0000 7f7342dc5640 1 -- 172.21.15.124:0/2579203587 wait complete. 2023-12-13T23:56:53.936 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:56:54.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.040+0000 7f7b9fa1e640 1 Processor -- start 2023-12-13T23:56:54.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.040+0000 7f7b9fa1e640 1 -- start start 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9fa1e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b980b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9fa1e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980b2b10 con 0x7f7b9815d330 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9fa1e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b980d1590 con 0x7f7b980d2910 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9fa1e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980d1870 con 0x7f7b980d3900 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9d7ba640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f7b9815d330 0x7f7b980b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50822/0 (socket says 172.21.15.124:50822) 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b9d7ba640 1 -- 172.21.15.124:0/2406761429 learned_addr learned my addr 172.21.15.124:0/2406761429 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:54.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.041+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134221307 0 0) 0x7f7b980b2b10 con 0x7f7b9815d330 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b84003520 con 0x7f7b9815d330 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9dfbb640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b980b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1984707780 0 0) 0x7f7b84003520 con 0x7f7b9815d330 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 msgr2=0x7f7b980b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b980b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 >> v1:172.21.15.203:6789/0 conn(0x7f7b980d2910 legacy=0x7f7b980d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980d1b50 con 0x7f7b9815d330 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f7b940019f0 con 0x7f7b9815d330 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 >> v1:172.21.15.124:6789/0 conn(0x7f7b9815d330 legacy=0x7f7b980b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b840039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b84003f20 0x7f7b840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b84004d00 0x7f7b84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980d1870 con 0x7f7b84003f20 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8400bb10 con 0x7f7b84004d00 2023-12-13T23:56:54.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8400bcd0 con 0x7f7b980d3900 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b84003f20 0x7f7b840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9d7ba640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b840039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9dfbb640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b84004d00 0x7f7b84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 -- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 msgr2=0x7f7b840039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b980d3900 0x7f7b840039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 -- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b84004d00 msgr2=0x7f7b84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b84004d00 0x7f7b84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.042+0000 7f7b9cfb9640 1 -- 172.21.15.124:0/2406761429 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b84003520 con 0x7f7b84003f20 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.043+0000 7f7b9cfb9640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b84003f20 0x7f7b840047c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7b88001750 tx=0x7f7b840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cabec14ab8105f7d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.043+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b880052d0 con 0x7f7b84003f20 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.043+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8802e050 con 0x7f7b84003f20 2023-12-13T23:56:54.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.043+0000 7f7b8ffff640 1 -- 172.21.15.124:0/2406761429 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b880035f0 con 0x7f7b84003f20 2023-12-13T23:56:54.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/2406761429 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b84003f20 msgr2=0x7f7b840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/2406761429 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b84003f20 0x7f7b840047c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f7b88001750 tx=0x7f7b840057f0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9dfbb640 1 -- 172.21.15.124:0/2406761429 reap_dead start 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/2406761429 shutdown_connections 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/2406761429 >> 172.21.15.124:0/2406761429 conn(0x7f7b98072020 msgr2=0x7f7b98072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/2406761429 shutdown_connections 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.044+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/2406761429 wait complete. 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.045+0000 7f7b9fa1e640 1 Processor -- start 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.045+0000 7f7b9fa1e640 1 -- start start 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b84003f20 0x7f7b98155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b98143310 0x7f7b981561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b98156aa0 con 0x7f7b980c81a0 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b981567c0 con 0x7f7b98143310 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b980c76c0 con 0x7f7b84003f20 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48432/0 (socket says 172.21.15.124:48432) 2023-12-13T23:56:54.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 -- 172.21.15.124:0/3735134678 learned_addr learned my addr 172.21.15.124:0/3735134678 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9d7ba640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b84003f20 0x7f7b98155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9cfb9640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b98143310 0x7f7b981561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 -- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b84003f20 msgr2=0x7f7b98155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b84003f20 0x7f7b98155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 -- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b98143310 msgr2=0x7f7b981561b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b98143310 0x7f7b981561b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 -- 172.21.15.124:0/3735134678 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b980c79a0 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9dfbb640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7b9808c510 tx=0x7f7b90011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b90011ee0 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b90004240 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b980c7c80 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.046+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9000f770 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.047+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b980c8b60 con 0x7f7b980c81a0 2023-12-13T23:56:54.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.048+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b980c7e70 con 0x7f7b980c81a0 2023-12-13T23:56:54.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.048+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7b9000f8d0 con 0x7f7b980c81a0 2023-12-13T23:56:54.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.048+0000 7f7b8dffb640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 0x7f7b7c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.049+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7b90089f50 con 0x7f7b980c81a0 2023-12-13T23:56:54.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.049+0000 7f7b9d7ba640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 0x7f7b7c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.049+0000 7f7b9d7ba640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 0x7f7b7c06d680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b94004d20 tx=0x7f7b94006040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.053+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7b90054c30 con 0x7f7b980c81a0 2023-12-13T23:56:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.219+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7b980693e0 con 0x7f7b980c81a0 2023-12-13T23:56:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.219+0000 7f7b8dffb640 1 -- 172.21.15.124:0/3735134678 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7b90000e60 con 0x7f7b980c81a0 2023-12-13T23:56:54.218 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:54.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 msgr2=0x7f7b7c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 0x7f7b7c06d680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b94004d20 tx=0x7f7b94006040 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 msgr2=0x7f7b980c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f7b9808c510 tx=0x7f7b90011cf0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 shutdown_connections 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b7c06b1c0 0x7f7b7c06d680 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b98143310 0x7f7b981561b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b980c81a0 0x7f7b980c8620 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 --2- 172.21.15.124:0/3735134678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b84003f20 0x7f7b98155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 >> 172.21.15.124:0/3735134678 conn(0x7f7b98072020 msgr2=0x7f7b980b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 shutdown_connections 2023-12-13T23:56:54.221 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.221+0000 7f7b9fa1e640 1 -- 172.21.15.124:0/3735134678 wait complete. 2023-12-13T23:56:54.231 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:56:54.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.333+0000 7f5727b68640 1 Processor -- start 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 -- start start 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f57200d2910 0x7f57200d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 0x7f57200b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200b2b10 con 0x7f572015d330 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57200d3bc0 con 0x7f57200d2910 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.334+0000 7f5727b68640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57200d1590 con 0x7f57200d3730 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f57200d2910 0x7f57200d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f57200d2910 0x7f57200d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48442/0 (socket says 172.21.15.124:48442) 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 -- 172.21.15.124:0/1279188279 learned_addr learned my addr 172.21.15.124:0/1279188279 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5726105640 1 --2- 172.21.15.124:0/1279188279 >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 0x7f57200b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 -- 172.21.15.124:0/1279188279 >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 msgr2=0x7f57200b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 --2- 172.21.15.124:0/1279188279 >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 0x7f57200b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 -- 172.21.15.124:0/1279188279 >> v1:172.21.15.203:6789/0 conn(0x7f572015d330 legacy=0x7f57200b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5725103640 1 -- 172.21.15.124:0/1279188279 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57200d1870 con 0x7f57200d2910 2023-12-13T23:56:54.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.335+0000 7f5726105640 1 --2- 172.21.15.124:0/1279188279 >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 0x7f57200b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:54.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5725103640 1 --2- 172.21.15.124:0/1279188279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d2910 0x7f57200d31f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f572006c500 tx=0x7f571002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e8ab11dacbaa230 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5724902640 1 -- 172.21.15.124:0/1279188279 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710031800 con 0x7f57200d2910 2023-12-13T23:56:54.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5724902640 1 -- 172.21.15.124:0/1279188279 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710031960 con 0x7f57200d2910 2023-12-13T23:56:54.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5724902640 1 -- 172.21.15.124:0/1279188279 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710031cf0 con 0x7f57200d2910 2023-12-13T23:56:54.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 -- 172.21.15.124:0/1279188279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d2910 msgr2=0x7f57200d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 --2- 172.21.15.124:0/1279188279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d2910 0x7f57200d31f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f572006c500 tx=0x7f571002b8a0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 -- 172.21.15.124:0/1279188279 shutdown_connections 2023-12-13T23:56:54.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 --2- 172.21.15.124:0/1279188279 >> v2:172.21.15.203:3300/0 conn(0x7f57200d3730 0x7f57200b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 --2- 172.21.15.124:0/1279188279 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d2910 0x7f57200d31f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 -- 172.21.15.124:0/1279188279 >> 172.21.15.124:0/1279188279 conn(0x7f5720072020 msgr2=0x7f5720072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 -- 172.21.15.124:0/1279188279 shutdown_connections 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.336+0000 7f5727b68640 1 -- 172.21.15.124:0/1279188279 wait complete. 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.337+0000 7f5727b68640 1 Processor -- start 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 -- start start 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57200d2910 0x7f57200cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 0x7f57200cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57200cc1f0 con 0x7f57200d3730 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572015fb10 con 0x7f572015d330 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5727b68640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572015fdf0 con 0x7f57200d2910 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5725103640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5725103640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48446/0 (socket says 172.21.15.124:48446) 2023-12-13T23:56:54.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.338+0000 7f5725103640 1 -- 172.21.15.124:0/313214431 learned_addr learned my addr 172.21.15.124:0/313214431 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5726105640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 0x7f57200cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 -- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57200d2910 msgr2=0x7f57200cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725904640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57200d2910 0x7f57200cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57200d2910 0x7f57200cbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 -- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 msgr2=0x7f57200cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 0x7f57200cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 -- 172.21.15.124:0/313214431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100094d0 con 0x7f57200d3730 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5726105640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 0x7f57200cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5725103640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5710000c00 tx=0x7f57100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f571003ad30 con 0x7f57200d3730 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571003ae90 con 0x7f57200d3730 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57200cb4b0 con 0x7f57200d3730 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5710044630 con 0x7f57200d3730 2023-12-13T23:56:54.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.339+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57201600d0 con 0x7f57200d3730 2023-12-13T23:56:54.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.340+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5720069400 con 0x7f57200d3730 2023-12-13T23:56:54.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.340+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f571004d450 con 0x7f57200d3730 2023-12-13T23:56:54.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.341+0000 7f57167fc640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 0x7f570406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.341+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f571002e070 con 0x7f57200d3730 2023-12-13T23:56:54.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.341+0000 7f5725904640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 0x7f570406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.341+0000 7f5725904640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 0x7f570406d6b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a750 tx=0x7f571c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.345+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5710076d20 con 0x7f57200d3730 2023-12-13T23:56:54.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.512+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f57200b46d0 con 0x7f57200d3730 2023-12-13T23:56:54.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.512+0000 7f57167fc640 1 -- 172.21.15.124:0/313214431 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7f57100350a0 con 0x7f57200d3730 2023-12-13T23:56:54.511 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:56:54.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 msgr2=0x7f570406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 0x7f570406d6b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f571c00a750 tx=0x7f571c008040 comp rx=0 tx=0).stop 2023-12-13T23:56:54.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 msgr2=0x7f5720154470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f5710000c00 tx=0x7f57100046c0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 shutdown_connections 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f570406b1f0 0x7f570406d6b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f572015d330 0x7f57200cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57200d3730 0x7f5720154470 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 --2- 172.21.15.124:0/313214431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57200d2910 0x7f57200cbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 >> 172.21.15.124:0/313214431 conn(0x7f5720072020 msgr2=0x7f5720072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 shutdown_connections 2023-12-13T23:56:54.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.514+0000 7f5727b68640 1 -- 172.21.15.124:0/313214431 wait complete. 2023-12-13T23:56:54.524 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:56:54.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 Processor -- start 2023-12-13T23:56:54.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 -- start start 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f60d40ceb70 0x7f60d40cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f60d40b4370 0x7f60d40b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d4071770 con 0x7f60d4144980 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d415ced0 con 0x7f60d40ceb70 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.627+0000 7f60d9f1d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d40b4ce0 con 0x7f60d40b4370 2023-12-13T23:56:54.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d2ffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f60d40ceb70 0x7f60d40cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d37fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f60d4144980 0x7f60d415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50836/0 (socket says 172.21.15.124:50836) 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d37fe640 1 -- 172.21.15.124:0/2976537359 learned_addr learned my addr 172.21.15.124:0/2976537359 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d3fff640 1 --2- 172.21.15.124:0/2976537359 >> v2:172.21.15.203:3300/0 conn(0x7f60d40b4370 0x7f60d40b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d2ffd640 1 -- 172.21.15.124:0/2976537359 >> v2:172.21.15.203:3300/0 conn(0x7f60d40b4370 msgr2=0x7f60d40b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d2ffd640 1 --2- 172.21.15.124:0/2976537359 >> v2:172.21.15.203:3300/0 conn(0x7f60d40b4370 0x7f60d40b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d2ffd640 1 -- 172.21.15.124:0/2976537359 >> v1:172.21.15.124:6789/0 conn(0x7f60d4144980 legacy=0x7f60d415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.628 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.628+0000 7f60d2ffd640 1 -- 172.21.15.124:0/2976537359 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40ccdc0 con 0x7f60d40ceb70 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d2ffd640 1 --2- 172.21.15.124:0/2976537359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40ceb70 0x7f60d40cc450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f60d40716e0 tx=0x7f60c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4792124072d348 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d27fc640 1 -- 172.21.15.124:0/2976537359 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8031830 con 0x7f60d40ceb70 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d27fc640 1 -- 172.21.15.124:0/2976537359 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c8031990 con 0x7f60d40ceb70 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d27fc640 1 -- 172.21.15.124:0/2976537359 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8031d20 con 0x7f60d40ceb70 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 -- 172.21.15.124:0/2976537359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40ceb70 msgr2=0x7f60d40cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/2976537359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40ceb70 0x7f60d40cc450 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f60d40716e0 tx=0x7f60c802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 -- 172.21.15.124:0/2976537359 shutdown_connections 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/2976537359 >> v2:172.21.15.203:3300/0 conn(0x7f60d40b4370 0x7f60d40b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/2976537359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40ceb70 0x7f60d40cc450 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 -- 172.21.15.124:0/2976537359 >> 172.21.15.124:0/2976537359 conn(0x7f60d406a4d0 msgr2=0x7f60d406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 -- 172.21.15.124:0/2976537359 shutdown_connections 2023-12-13T23:56:54.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.629+0000 7f60d9f1d640 1 -- 172.21.15.124:0/2976537359 wait complete. 2023-12-13T23:56:54.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.630+0000 7f60d9f1d640 1 Processor -- start 2023-12-13T23:56:54.631 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 -- start start 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d40ceb70 0x7f60d40b3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60d4144980 0x7f60d40b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40c88b0 con 0x7f60d40b4370 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40b2990 con 0x7f60d40ceb70 2023-12-13T23:56:54.633 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d9f1d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d40b2c70 con 0x7f60d4144980 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48464/0 (socket says 172.21.15.124:48464) 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.631+0000 7f60d37fe640 1 -- 172.21.15.124:0/1314132746 learned_addr learned my addr 172.21.15.124:0/1314132746 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 -- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60d4144980 msgr2=0x7f60d40b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60d4144980 0x7f60d40b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 -- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d40ceb70 msgr2=0x7f60d40b3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d40ceb70 0x7f60d40b3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 -- 172.21.15.124:0/1314132746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c8009400 con 0x7f60d40b4370 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d37fe640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f60c400cbc0 tx=0x7f60c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c400aca0 con 0x7f60d40b4370 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c400ae00 con 0x7f60d40b4370 2023-12-13T23:56:54.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c4012600 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d40b2f50 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.632+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d40b26a0 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.633+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d40c8d40 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.633+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f60c4007070 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.634+0000 7f60bbfff640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 0x7f60c006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.634+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f60c4089960 con 0x7f60d40b4370 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.634+0000 7f60d2ffd640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 0x7f60c006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.635+0000 7f60d2ffd640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 0x7f60c006d680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f60c8036000 tx=0x7f60c8030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.638+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f60d40c8d40 con 0x7f60d40b4370 2023-12-13T23:56:54.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.804+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f60d40c8d40 con 0x7f60d40b4370 2023-12-13T23:56:54.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.804+0000 7f60bbfff640 1 -- 172.21.15.124:0/1314132746 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f60c4017020 con 0x7f60d40b4370 2023-12-13T23:56:54.803 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:56:54.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 msgr2=0x7f60c006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 0x7f60c006d680 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f60c8036000 tx=0x7f60c8030000 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 msgr2=0x7f60d4140cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f60c400cbc0 tx=0x7f60c400a060 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 shutdown_connections 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f60c006b1c0 0x7f60c006d680 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f60d40ceb70 0x7f60d40b3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f60d4144980 0x7f60d40b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 --2- 172.21.15.124:0/1314132746 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f60d40b4370 0x7f60d4140cd0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 >> 172.21.15.124:0/1314132746 conn(0x7f60d406a4d0 msgr2=0x7f60d41590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 shutdown_connections 2023-12-13T23:56:54.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.806+0000 7f60d9f1d640 1 -- 172.21.15.124:0/1314132746 wait complete. 2023-12-13T23:56:54.816 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:56:54.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.919+0000 7ff51ef83640 1 Processor -- start 2023-12-13T23:56:54.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.919+0000 7ff51ef83640 1 -- start start 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51ef83640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff5180ceb70 0x7ff5180cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51ef83640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff5180b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51ef83640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518071770 con 0x7ff518144980 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51ef83640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51815ced0 con 0x7ff5180ceb70 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51ef83640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180b4ce0 con 0x7ff5180b4370 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff5180ceb70 0x7ff5180cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff5180ceb70 0x7ff5180cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48472/0 (socket says 172.21.15.124:48472) 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 -- 172.21.15.124:0/2373834254 learned_addr learned my addr 172.21.15.124:0/2373834254 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff51d520640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff5180b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 -- 172.21.15.124:0/2373834254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 msgr2=0x7ff5180b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff5180b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 -- 172.21.15.124:0/2373834254 >> v1:172.21.15.203:6789/0 conn(0x7ff518144980 legacy=0x7ff51815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:56:54.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.920+0000 7ff517fff640 1 -- 172.21.15.124:0/2373834254 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5180ccdc0 con 0x7ff5180ceb70 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.921+0000 7ff517fff640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5180ceb70 0x7ff5180cc450 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397b2ae7248f9bae server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.921+0000 7ff5177fe640 1 -- 172.21.15.124:0/2373834254 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504031640 con 0x7ff5180ceb70 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.921+0000 7ff5177fe640 1 -- 172.21.15.124:0/2373834254 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5040317a0 con 0x7ff5180ceb70 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.921+0000 7ff5177fe640 1 -- 172.21.15.124:0/2373834254 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504031b30 con 0x7ff5180ceb70 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 -- 172.21.15.124:0/2373834254 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5180ceb70 msgr2=0x7ff5180cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5180ceb70 0x7ff5180cc450 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7ff504009870 tx=0x7ff50402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 -- 172.21.15.124:0/2373834254 shutdown_connections 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff5180b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 --2- 172.21.15.124:0/2373834254 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff5180ceb70 0x7ff5180cc450 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 -- 172.21.15.124:0/2373834254 >> 172.21.15.124:0/2373834254 conn(0x7ff51806a4d0 msgr2=0x7ff51806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 -- 172.21.15.124:0/2373834254 shutdown_connections 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.922+0000 7ff51ef83640 1 -- 172.21.15.124:0/2373834254 wait complete. 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.923+0000 7ff51ef83640 1 Processor -- start 2023-12-13T23:56:54.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.923+0000 7ff51ef83640 1 -- start start 2023-12-13T23:56:54.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff518154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5180ceb70 0x7ff518158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180b2c10 con 0x7ff518144980 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180b2eb0 con 0x7ff5180ceb70 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51ef83640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180b3190 con 0x7ff5180b4370 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:48486/0 (socket says 172.21.15.124:48486) 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 -- 172.21.15.124:0/1712928342 learned_addr learned my addr 172.21.15.124:0/1712928342 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff517fff640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5180ceb70 0x7ff518158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51cd1f640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff518154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 -- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 msgr2=0x7ff518154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff518154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 -- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5180ceb70 msgr2=0x7ff518158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5180ceb70 0x7ff518158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:54.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.924+0000 7ff51d520640 1 -- 172.21.15.124:0/1712928342 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040094d0 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff51d520640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff510007e60 tx=0x7ff51000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510014420 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5100145a0 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5180b3470 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510014950 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.925+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5180b3c70 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.926+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff51815b4c0 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.926+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff510014ab0 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.926+0000 7ff5157fa640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 0x7ff4fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.927+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff51008a590 con 0x7ff518144980 2023-12-13T23:56:54.927 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.927+0000 7ff51cd1f640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 0x7ff4fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:56:54.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.927+0000 7ff51cd1f640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 0x7ff4fc06d680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff50c004ea0 tx=0x7ff50c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:56:54.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:54.931+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff5100550a0 con 0x7ff518144980 2023-12-13T23:56:55.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.110+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff5180b3990 con 0x7ff518144980 2023-12-13T23:56:55.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.110+0000 7ff5157fa640 1 -- 172.21.15.124:0/1712928342 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v38) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff5180b3990 con 0x7ff518144980 2023-12-13T23:56:55.109 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:56:55.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 msgr2=0x7ff4fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:55.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 0x7ff4fc06d680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff50c004ea0 tx=0x7ff50c009040 comp rx=0 tx=0).stop 2023-12-13T23:56:55.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 msgr2=0x7ff5180c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:56:55.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7ff510007e60 tx=0x7ff51000e4b0 comp rx=0 tx=0).stop 2023-12-13T23:56:55.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 shutdown_connections 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff4fc06b1c0 0x7ff4fc06d680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff5180ceb70 0x7ff518158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff518144980 0x7ff5180c9ed0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 --2- 172.21.15.124:0/1712928342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff5180b4370 0x7ff518154a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 >> 172.21.15.124:0/1712928342 conn(0x7ff51806a4d0 msgr2=0x7ff51815e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 shutdown_connections 2023-12-13T23:56:55.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:56:55.112+0000 7ff51ef83640 1 -- 172.21.15.124:0/1712928342 wait complete. 2023-12-13T23:56:55.121 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:56:55.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:56:55.122 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:56:55.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[105784]: fuse finished with error 0 and tester_r 0 2023-12-13T23:56:55.149 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:01.250 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:01.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:01.250 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:57:01.269 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2023-12-13T23:57:01.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.428+0000 7f735a477640 1 Processor -- start 2023-12-13T23:57:01.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 -- start start 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f73540d3850 0x7f73540cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540d00f0 con 0x7f73540b43a0 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540cff00 con 0x7f73540d3850 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.429+0000 7f735a477640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540d1420 con 0x7f735415d420 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7358a14640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f73540d3850 0x7f73540cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:36262/0 (socket says 172.21.15.124:36262) 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 -- 172.21.15.124:0/2152747846 learned_addr learned my addr 172.21.15.124:0/2152747846 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f73537fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f73540b43a0 0x7f73540d3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:45920/0 (socket says 172.21.15.124:45920) 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 -- 172.21.15.124:0/2152747846 >> v2:172.21.15.203:3300/0 conn(0x7f73540d3850 msgr2=0x7f73540cf9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 --2- 172.21.15.124:0/2152747846 >> v2:172.21.15.203:3300/0 conn(0x7f73540d3850 0x7f73540cf9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 -- 172.21.15.124:0/2152747846 >> v1:172.21.15.203:6789/0 conn(0x7f73540b43a0 legacy=0x7f73540d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.430+0000 7f7353fff640 1 -- 172.21.15.124:0/2152747846 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73540d03d0 con 0x7f735415d420 2023-12-13T23:57:01.431 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.431+0000 7f7353fff640 1 --2- 172.21.15.124:0/2152747846 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7344009870 tx=0x7f734402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b12344e04085c26 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:01.431 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.431+0000 7f7352ffd640 1 -- 172.21.15.124:0/2152747846 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344030840 con 0x7f735415d420 2023-12-13T23:57:01.431 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.431+0000 7f7352ffd640 1 -- 172.21.15.124:0/2152747846 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73440309a0 con 0x7f735415d420 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.431+0000 7f7352ffd640 1 -- 172.21.15.124:0/2152747846 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344030d30 con 0x7f735415d420 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 -- 172.21.15.124:0/2152747846 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 msgr2=0x7f73540b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 --2- 172.21.15.124:0/2152747846 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7344009870 tx=0x7f734402b980 comp rx=0 tx=0).stop 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 -- 172.21.15.124:0/2152747846 shutdown_connections 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 --2- 172.21.15.124:0/2152747846 >> v2:172.21.15.203:3300/0 conn(0x7f73540d3850 0x7f73540cf9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 --2- 172.21.15.124:0/2152747846 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f735415d420 0x7f73540b2b50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 -- 172.21.15.124:0/2152747846 >> 172.21.15.124:0/2152747846 conn(0x7f735408b000 msgr2=0x7f735409c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 -- 172.21.15.124:0/2152747846 shutdown_connections 2023-12-13T23:57:01.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.432+0000 7f735a477640 1 -- 172.21.15.124:0/2152747846 wait complete. 2023-12-13T23:57:01.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.433+0000 7f735a477640 1 Processor -- start 2023-12-13T23:57:01.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.433+0000 7f735a477640 1 -- start start 2023-12-13T23:57:01.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73540d3850 0x7f73540c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f735415d420 0x7f73540c7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540c97a0 con 0x7f73540b43a0 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73540c8e70 con 0x7f735415d420 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f735a477640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540c9150 con 0x7f73540d3850 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57776/0 (socket says 172.21.15.124:57776) 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 -- 172.21.15.124:0/960814118 learned_addr learned my addr 172.21.15.124:0/960814118 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7358a14640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f735415d420 0x7f73540c7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f73537fe640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73540d3850 0x7f73540c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 -- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73540d3850 msgr2=0x7f73540c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73540d3850 0x7f73540c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 -- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f735415d420 msgr2=0x7f73540c7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f735415d420 0x7f73540c7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.434+0000 7f7353fff640 1 -- 172.21.15.124:0/960814118 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440094d0 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f7353fff640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f734402b9c0 tx=0x7f73440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734403e070 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7344039800 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73540c9430 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7344039bb0 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.435+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73540c8a90 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.436+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73540c95c0 con 0x7f73540b43a0 2023-12-13T23:57:01.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.436+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7344042070 con 0x7f73540b43a0 2023-12-13T23:57:01.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.437+0000 7f7350ff9640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 0x7f734006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:01.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.437+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f73440abba0 con 0x7f73540b43a0 2023-12-13T23:57:01.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.437+0000 7f73537fe640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 0x7f734006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:01.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.437+0000 7f73537fe640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 0x7f734006d680 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f735414aa10 tx=0x7f7348007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:01.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.441+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f73440766b0 con 0x7f73540b43a0 2023-12-13T23:57:01.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.669+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7354069400 con 0x7f73540b43a0 2023-12-13T23:57:01.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.671+0000 7f7350ff9640 1 -- 172.21.15.124:0/960814118 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) v1 ==== 184+0+141 (secure 0 0 0) 0x7f7344049020 con 0x7f73540b43a0 2023-12-13T23:57:01.670 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:57:01.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:57:01.671 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:57:01.671 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:57:01.671 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw pool=cephfs_data" 2023-12-13T23:57:01.671 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 msgr2=0x7f734006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 0x7f734006d680 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f735414aa10 tx=0x7f7348007c80 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 msgr2=0x7f735415fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f734402b9c0 tx=0x7f73440046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 shutdown_connections 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f734006b1c0 0x7f734006d680 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f735415d420 0x7f73540c7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f73540d3850 0x7f73540c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.673+0000 7f735a477640 1 --2- 172.21.15.124:0/960814118 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f73540b43a0 0x7f735415fe40 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.674+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 >> 172.21.15.124:0/960814118 conn(0x7f735408b000 msgr2=0x7f735406b320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.674+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 shutdown_connections 2023-12-13T23:57:01.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:01.674+0000 7f735a477640 1 -- 172.21.15.124:0/960814118 wait complete. 2023-12-13T23:57:01.683 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-13T23:57:01.684 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:01.724 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:01.770 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-13T23:57:01.771 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:01.771 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:01.786 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:01.786 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:01.786 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:01.786 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:57:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:01.842 INFO:teuthology.orchestra.run.smithi124.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-13T23:57:01.842 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:01.842 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:01.861 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:01.861 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:57:01.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:01.861 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-13T23:57:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:01.919 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-13T23:57:01.919 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:01.919 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:01.935 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:01.935 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:01.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:01.935 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-13T23:57:01.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:01.989 INFO:teuthology.orchestra.run.smithi124.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-13T23:57:01.990 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:01.990 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:02.008 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:02.008 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:57:02.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:02.008 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-13T23:57:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:02.065 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-13T23:57:02.065 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:02.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.166+0000 7f08d7583640 1 Processor -- start 2023-12-13T23:57:02.165 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.166+0000 7f08d7583640 1 -- start start 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d7583640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d0150940 0x7f08d015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d7583640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f08d015dec0 0x7f08d015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d7583640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08d00b49f0 con 0x7f08d0150560 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d7583640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08d00b3560 con 0x7f08d015dec0 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d7583640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00b3de0 con 0x7f08d0150940 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d531f640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f08d0150560 0x7f08d00b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41948/0 (socket says 172.21.15.124:41948) 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d531f640 1 -- 172.21.15.124:0/1894167359 learned_addr learned my addr 172.21.15.124:0/1894167359 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128964125 0 0) 0x7f08d00b49f0 con 0x7f08d0150560 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08bc003520 con 0x7f08d0150560 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d0150940 0x7f08d015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.169 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> v2:172.21.15.203:3300/0 conn(0x7f08d015dec0 0x7f08d015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3645237862 0 0) 0x7f08bc003520 con 0x7f08d0150560 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d0150940 msgr2=0x7f08d015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d0150940 0x7f08d015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 >> v2:172.21.15.203:3300/0 conn(0x7f08d015dec0 msgr2=0x7f08d015d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 --2- 172.21.15.124:0/1894167359 >> v2:172.21.15.203:3300/0 conn(0x7f08d015dec0 0x7f08d015d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d00b40c0 con 0x7f08d0150560 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f08cc001670 con 0x7f08d0150560 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> v2:172.21.15.203:3300/0 conn(0x7f08d015dec0 0x7f08d015d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d0150940 0x7f08d015d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 >> v1:172.21.15.124:6789/0 conn(0x7f08d0150560 legacy=0x7f08d00b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc003970 0x7f08bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08bc0047c0 0x7f08bc004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.170 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00b3560 con 0x7f08bc005100 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d00b3de0 con 0x7f08bc003970 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.167+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc00e0d0 con 0x7f08bc0047c0 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d531f640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc003970 0x7f08bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08bc0047c0 0x7f08bc004bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 -- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08bc0047c0 msgr2=0x7f08bc004bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08bc0047c0 0x7f08bc004bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 -- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc003970 msgr2=0x7f08bc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08bc003970 0x7f08bc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d5b20640 1 -- 172.21.15.124:0/1894167359 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc003520 con 0x7f08bc005100 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.168+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08bc0047c0 0x7f08bc004bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d5b20640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08bc00b980 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f08c80073b0 tx=0x7f08bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e6a88d9eeecf18 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c802cd90 con 0x7f08bc005100 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c8003990 con 0x7f08bc005100 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08c7fff640 1 -- 172.21.15.124:0/1894167359 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c8003d00 con 0x7f08bc005100 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 -- 172.21.15.124:0/1894167359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 msgr2=0x7f08bc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 --2- 172.21.15.124:0/1894167359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08bc00b980 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f08c80073b0 tx=0x7f08bc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d5b20640 1 -- 172.21.15.124:0/1894167359 reap_dead start 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 -- 172.21.15.124:0/1894167359 shutdown_connections 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 -- 172.21.15.124:0/1894167359 >> 172.21.15.124:0/1894167359 conn(0x7f08d006a4d0 msgr2=0x7f08d006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 -- 172.21.15.124:0/1894167359 shutdown_connections 2023-12-13T23:57:02.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.169+0000 7f08d7583640 1 -- 172.21.15.124:0/1894167359 wait complete. 2023-12-13T23:57:02.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.170+0000 7f08d7583640 1 Processor -- start 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 -- start start 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 0x7f08d015adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 0x7f08d0157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d0143630 con 0x7f08bc005100 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08d015b800 con 0x7f08d01431b0 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d7583640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08d015bae0 con 0x7f08d015b2f0 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d531f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d531f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57808/0 (socket says 172.21.15.124:57808) 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.171+0000 7f08d531f640 1 -- 172.21.15.124:0/21551422 learned_addr learned my addr 172.21.15.124:0/21551422 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 0x7f08d015adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.177 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d5b20640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 0x7f08d0157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 -- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 msgr2=0x7f08d0157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 0x7f08d0157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 -- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 msgr2=0x7f08d015adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 0x7f08d015adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 -- 172.21.15.124:0/21551422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d015bdc0 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d5b20640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 0x7f08d0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 0x7f08d015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d531f640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f08cc007340 tx=0x7f08cc007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00f8e0 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08cc00fa80 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08d015c0a0 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08cc00fe30 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.172+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08d0143a20 con 0x7f08bc005100 2023-12-13T23:57:02.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.173+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08d0069400 con 0x7f08bc005100 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.174+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f08cc01d460 con 0x7f08bc005100 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.174+0000 7f08c5ffb640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 0x7f08b406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.174+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f08cc089d60 con 0x7f08bc005100 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.174+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 0x7f08b406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.175+0000 7f08d4b1e640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 0x7f08b406d680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f08d01440b0 tx=0x7f08c0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.179+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f08cc054990 con 0x7f08bc005100 2023-12-13T23:57:02.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.377+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08d014b150 con 0x7f08bc005100 2023-12-13T23:57:02.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.377+0000 7f08c5ffb640 1 -- 172.21.15.124:0/21551422 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4923 (secure 0 0 0) 0x7f08cc057fb0 con 0x7f08bc005100 2023-12-13T23:57:02.377 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:02.377 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:26.366668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:57:02.377 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 15 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 msgr2=0x7f08b406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 0x7f08b406d680 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f08d01440b0 tx=0x7f08c0008040 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 msgr2=0x7f08d015c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f08cc007340 tx=0x7f08cc007800 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 shutdown_connections 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f08b406b1c0 0x7f08b406d680 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f08d01431b0 0x7f08d015adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f08d015b2f0 0x7f08d0157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 --2- 172.21.15.124:0/21551422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f08bc005100 0x7f08d015c8a0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.379+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 >> 172.21.15.124:0/21551422 conn(0x7f08d006a4d0 msgr2=0x7f08d00ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.380+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 shutdown_connections 2023-12-13T23:57:02.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.380+0000 7f08d7583640 1 -- 172.21.15.124:0/21551422 wait complete. 2023-12-13T23:57:02.390 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:02.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.490+0000 7f67a91dc640 1 Processor -- start 2023-12-13T23:57:02.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.490+0000 7f67a91dc640 1 -- start start 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a91dc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f67a4144980 0x7f67a415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a91dc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f67a40b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a91dc640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67a4071770 con 0x7f67a40ceb70 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a91dc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67a40ccb60 con 0x7f67a4144980 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a91dc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40b4ce0 con 0x7f67a40b4370 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f679bfff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f67a40ceb70 0x7f67a40cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41964/0 (socket says 172.21.15.124:41964) 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f679bfff640 1 -- 172.21.15.124:0/2195317608 learned_addr learned my addr 172.21.15.124:0/2195317608 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a2d9d640 1 --2- 172.21.15.124:0/2195317608 >> v2:172.21.15.203:3300/0 conn(0x7f67a4144980 0x7f67a415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840533475 0 0) 0x7f67a4071770 con 0x7f67a40ceb70 2023-12-13T23:57:02.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6790003520 con 0x7f67a40ceb70 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a359e640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f67a40b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658712058 0 0) 0x7f6790003520 con 0x7f67a40ceb70 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 msgr2=0x7f67a40b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f67a40b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 >> v2:172.21.15.203:3300/0 conn(0x7f67a4144980 msgr2=0x7f67a415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 --2- 172.21.15.124:0/2195317608 >> v2:172.21.15.203:3300/0 conn(0x7f67a4144980 0x7f67a415c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40ccdc0 con 0x7f67a40ceb70 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f678c001670 con 0x7f67a40ceb70 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 >> v1:172.21.15.124:6789/0 conn(0x7f67a40ceb70 legacy=0x7f67a40cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f6790003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6790003ef0 0x7f67900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6790004d00 0x7f6790005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40ccb60 con 0x7f6790004d00 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40b4ce0 con 0x7f6790003ef0 2023-12-13T23:57:02.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.491+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679000b980 con 0x7f67a40b4370 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6790004d00 0x7f6790005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a2d9d640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f6790003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f679bfff640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6790003ef0 0x7f67900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 -- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 msgr2=0x7f6790003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40b4370 0x7f6790003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 -- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6790003ef0 msgr2=0x7f67900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6790003ef0 0x7f67900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.492+0000 7f67a359e640 1 -- 172.21.15.124:0/2195317608 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6790003520 con 0x7f6790004d00 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a359e640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6790004d00 0x7f6790005180 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f679c009fc0 tx=0x7f67900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1313883282578b51 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c02cc00 con 0x7f6790004d00 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c00f040 con 0x7f6790004d00 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a259c640 1 -- 172.21.15.124:0/2195317608 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c034440 con 0x7f6790004d00 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 -- 172.21.15.124:0/2195317608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6790004d00 msgr2=0x7f6790005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 --2- 172.21.15.124:0/2195317608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6790004d00 0x7f6790005180 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f679c009fc0 tx=0x7f67900057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a359e640 1 -- 172.21.15.124:0/2195317608 reap_dead start 2023-12-13T23:57:02.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 -- 172.21.15.124:0/2195317608 shutdown_connections 2023-12-13T23:57:02.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 -- 172.21.15.124:0/2195317608 >> 172.21.15.124:0/2195317608 conn(0x7f67a406a4d0 msgr2=0x7f67a406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 -- 172.21.15.124:0/2195317608 shutdown_connections 2023-12-13T23:57:02.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.493+0000 7f67a91dc640 1 -- 172.21.15.124:0/2195317608 wait complete. 2023-12-13T23:57:02.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 Processor -- start 2023-12-13T23:57:02.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 -- start start 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a4154ab0 0x7f67a4135cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40c6e10 0x7f67a40c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40c40e0 con 0x7f67a40c7ce0 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a4154e90 con 0x7f67a4154ab0 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a91dc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67a40c4350 con 0x7f67a40c6e10 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a359e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a359e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57822/0 (socket says 172.21.15.124:57822) 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a359e640 1 -- 172.21.15.124:0/4203090284 learned_addr learned my addr 172.21.15.124:0/4203090284 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f67a2d9d640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a4154ab0 0x7f67a4135cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.495+0000 7f679bfff640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40c6e10 0x7f67a40c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 -- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40c6e10 msgr2=0x7f67a40c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40c6e10 0x7f67a40c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 -- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a4154ab0 msgr2=0x7f67a4135cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a4154ab0 0x7f67a4135cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 -- 172.21.15.124:0/4203090284 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a40c4630 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a359e640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f679c037040 tx=0x7f679c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c034d50 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c02de90 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a40c4910 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c003bd0 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.496+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a40c7290 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.497+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a40c4e30 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.497+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f679c003d30 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.498+0000 7f679b7fe640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 0x7f678406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.498+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f679c0ab340 con 0x7f67a40c7ce0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.498+0000 7f67a2d9d640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 0x7f678406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.498+0000 7f67a2d9d640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 0x7f678406d6b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6794009b40 tx=0x7f6794008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.502+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f67a40c4e30 con 0x7f67a40c7ce0 2023-12-13T23:57:02.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.671+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67a40c4e30 con 0x7f67a40c7ce0 2023-12-13T23:57:02.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.672+0000 7f679b7fe640 1 -- 172.21.15.124:0/4203090284 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f67a40c4e30 con 0x7f67a40c7ce0 2023-12-13T23:57:02.671 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:02.671 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:02.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 msgr2=0x7f678406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 0x7f678406d6b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6794009b40 tx=0x7f6794008040 comp rx=0 tx=0).stop 2023-12-13T23:57:02.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 msgr2=0x7f67a40c5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f679c037040 tx=0x7f679c009750 comp rx=0 tx=0).stop 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 shutdown_connections 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f678406b1f0 0x7f678406d6b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f67a40c7ce0 0x7f67a40c5110 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f67a40c6e10 0x7f67a40c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 --2- 172.21.15.124:0/4203090284 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f67a4154ab0 0x7f67a4135cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 >> 172.21.15.124:0/4203090284 conn(0x7f67a406a4d0 msgr2=0x7f67a41590b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 shutdown_connections 2023-12-13T23:57:02.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.674+0000 7f67a91dc640 1 -- 172.21.15.124:0/4203090284 wait complete. 2023-12-13T23:57:02.684 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:56:16.668961+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:56:17.292190+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '35', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2023-12-13T23:56:25.305690+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '38', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:02.685 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:02.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.789+0000 7f06deff1640 1 Processor -- start 2023-12-13T23:57:02.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.790+0000 7f06deff1640 1 -- start start 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.790+0000 7f06deff1640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f06d815d330 0x7f06d80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.790+0000 7f06deff1640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d80b2b10 con 0x7f06d80d3900 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.790+0000 7f06deff1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06d80d1590 con 0x7f06d80d2910 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.790+0000 7f06deff1640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06d80d1870 con 0x7f06d815d330 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f06d815d330 0x7f06d80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f06d815d330 0x7f06d80b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57836/0 (socket says 172.21.15.124:57836) 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 -- 172.21.15.124:0/1298481976 learned_addr learned my addr 172.21.15.124:0/1298481976 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dd58e640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f06d80d3900 0x7f06d80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41968/0 (socket says 172.21.15.124:41968) 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 -- 172.21.15.124:0/1298481976 >> v1:172.21.15.203:6789/0 conn(0x7f06d80d2910 legacy=0x7f06d80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 -- 172.21.15.124:0/1298481976 >> v1:172.21.15.124:6789/0 conn(0x7f06d80d3900 legacy=0x7f06d80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06d77fe640 1 -- 172.21.15.124:0/1298481976 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667275438 0 0) 0x7f06d80b2b10 con 0x7f06d80d3900 2023-12-13T23:57:02.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.791+0000 7f06dcd8d640 1 -- 172.21.15.124:0/1298481976 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d80d1b50 con 0x7f06d815d330 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06dcd8d640 1 --2- 172.21.15.124:0/1298481976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d815d330 0x7f06d80b8b80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009870 tx=0x7f06cc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abab47f9c55a37e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06d77fe640 1 -- 172.21.15.124:0/1298481976 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc030860 con 0x7f06d815d330 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06d77fe640 1 -- 172.21.15.124:0/1298481976 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06cc0309c0 con 0x7f06d815d330 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06d77fe640 1 -- 172.21.15.124:0/1298481976 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06cc030d50 con 0x7f06d815d330 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 -- 172.21.15.124:0/1298481976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d815d330 msgr2=0x7f06d80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 --2- 172.21.15.124:0/1298481976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d815d330 0x7f06d80b8b80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f06cc009870 tx=0x7f06cc02b9a0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 -- 172.21.15.124:0/1298481976 shutdown_connections 2023-12-13T23:57:02.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 --2- 172.21.15.124:0/1298481976 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d815d330 0x7f06d80b8b80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 -- 172.21.15.124:0/1298481976 >> 172.21.15.124:0/1298481976 conn(0x7f06d8072020 msgr2=0x7f06d8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 -- 172.21.15.124:0/1298481976 shutdown_connections 2023-12-13T23:57:02.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.792+0000 7f06deff1640 1 -- 172.21.15.124:0/1298481976 wait complete. 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.793+0000 7f06deff1640 1 Processor -- start 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.793+0000 7f06deff1640 1 -- start start 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f06d80d2910 0x7f06d814c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06d815d330 0x7f06d80c7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d815fd30 con 0x7f06d80d3900 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d8160010 con 0x7f06d815d330 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06deff1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d81602f0 con 0x7f06d80d2910 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06d7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06d7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57840/0 (socket says 172.21.15.124:57840) 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.794+0000 7f06d7fff640 1 -- 172.21.15.124:0/3330073152 learned_addr learned my addr 172.21.15.124:0/3330073152 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06dd58e640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06d815d330 0x7f06d80c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06dcd8d640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f06d80d2910 0x7f06d814c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 -- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f06d80d2910 msgr2=0x7f06d814c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f06d80d2910 0x7f06d814c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 -- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06d815d330 msgr2=0x7f06d80c7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06d815d330 0x7f06d80c7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 -- 172.21.15.124:0/3330073152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06cc0094d0 con 0x7f06d80d3900 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d7fff640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c450 tx=0x7f06c400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c40052d0 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06c4005430 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06d81605d0 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06c4012870 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.795+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06d814cab0 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.796+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d80d0b50 con 0x7f06d80d3900 2023-12-13T23:57:02.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.797+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f06c40129d0 con 0x7f06d80d3900 2023-12-13T23:57:02.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.797+0000 7f06d57fa640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 0x7f06bc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:02.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.797+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f06c40895c0 con 0x7f06d80d3900 2023-12-13T23:57:02.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.797+0000 7f06dcd8d640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 0x7f06bc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:02.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.798+0000 7f06dcd8d640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 0x7f06bc06d680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06cc036000 tx=0x7f06cc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:02.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.801+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f06c40541f0 con 0x7f06d80d3900 2023-12-13T23:57:02.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.968+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f06d8067700 con 0x7f06d80d3900 2023-12-13T23:57:02.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.969+0000 7f06d57fa640 1 -- 172.21.15.124:0/3330073152 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f06c4057810 con 0x7f06d80d3900 2023-12-13T23:57:02.968 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.970+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 msgr2=0x7f06bc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.970+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 0x7f06bc06d680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f06cc036000 tx=0x7f06cc008030 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.970+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 msgr2=0x7f06d80c9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.970+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f06c400c450 tx=0x7f06c400c910 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 shutdown_connections 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f06bc06b1c0 0x7f06bc06d680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f06d815d330 0x7f06d80c7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f06d80d3900 0x7f06d80c9340 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 --2- 172.21.15.124:0/3330073152 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f06d80d2910 0x7f06d814c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:02.970 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 >> 172.21.15.124:0/3330073152 conn(0x7f06d8072020 msgr2=0x7f06d80d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:02.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 shutdown_connections 2023-12-13T23:57:02.971 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:02.971+0000 7f06deff1640 1 -- 172.21.15.124:0/3330073152 wait complete. 2023-12-13T23:57:02.980 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:03.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.084+0000 7fb16dfe5640 1 Processor -- start 2023-12-13T23:57:03.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.084+0000 7fb16dfe5640 1 -- start start 2023-12-13T23:57:03.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb16dfe5640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb16815d1d0 0x7fb16806b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb16dfe5640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1680aeb80 0x7fb16806ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb16dfe5640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1680adae0 con 0x7fb1680ad640 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb16dfe5640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680addc0 con 0x7fb1680aeb80 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb16dfe5640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1680ae0a0 con 0x7fb16815d1d0 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb167fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1680aeb80 0x7fb16806ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb167fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb1680aeb80 0x7fb16806ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57842/0 (socket says 172.21.15.124:57842) 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.085+0000 7fb167fff640 1 -- 172.21.15.124:0/3688851249 learned_addr learned my addr 172.21.15.124:0/3688851249 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb1677fe640 1 --2- 172.21.15.124:0/3688851249 >> v2:172.21.15.203:3300/0 conn(0x7fb16815d1d0 0x7fb16806b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb167fff640 1 -- 172.21.15.124:0/3688851249 >> v2:172.21.15.203:3300/0 conn(0x7fb16815d1d0 msgr2=0x7fb16806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb167fff640 1 --2- 172.21.15.124:0/3688851249 >> v2:172.21.15.203:3300/0 conn(0x7fb16815d1d0 0x7fb16806b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb167fff640 1 -- 172.21.15.124:0/3688851249 >> v1:172.21.15.203:6789/0 conn(0x7fb1680ad640 legacy=0x7fb168160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb167fff640 1 -- 172.21.15.124:0/3688851249 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1680ae380 con 0x7fb1680aeb80 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb167fff640 1 --2- 172.21.15.124:0/3688851249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680aeb80 0x7fb16806ba20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb168072310 tx=0x7fb16002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69315172a4f260b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.086+0000 7fb1667fc640 1 -- 172.21.15.124:0/3688851249 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16002ca80 con 0x7fb1680aeb80 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb1667fc640 1 -- 172.21.15.124:0/3688851249 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16002cbe0 con 0x7fb1680aeb80 2023-12-13T23:57:03.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb1667fc640 1 -- 172.21.15.124:0/3688851249 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1600343f0 con 0x7fb1680aeb80 2023-12-13T23:57:03.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 -- 172.21.15.124:0/3688851249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680aeb80 msgr2=0x7fb16806ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/3688851249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680aeb80 0x7fb16806ba20 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fb168072310 tx=0x7fb16002bee0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 -- 172.21.15.124:0/3688851249 shutdown_connections 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/3688851249 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680aeb80 0x7fb16806ba20 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/3688851249 >> v2:172.21.15.203:3300/0 conn(0x7fb16815d1d0 0x7fb16806b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 -- 172.21.15.124:0/3688851249 >> 172.21.15.124:0/3688851249 conn(0x7fb168070e70 msgr2=0x7fb168071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 -- 172.21.15.124:0/3688851249 shutdown_connections 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.087+0000 7fb16dfe5640 1 -- 172.21.15.124:0/3688851249 wait complete. 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.088+0000 7fb16dfe5640 1 Processor -- start 2023-12-13T23:57:03.088 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.088+0000 7fb16dfe5640 1 -- start start 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1680aeb80 0x7fb1681602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb16815d1d0 0x7fb1681f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1681f6f00 con 0x7fb1680ad640 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1681f6660 con 0x7fb16815d1d0 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb16dfe5640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1681f6940 con 0x7fb1680aeb80 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57854/0 (socket says 172.21.15.124:57854) 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 -- 172.21.15.124:0/2568639369 learned_addr learned my addr 172.21.15.124:0/2568639369 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.090 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 -- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1680aeb80 msgr2=0x7fb1681602e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1680aeb80 0x7fb1681602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 -- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb16815d1d0 msgr2=0x7fb1681f22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb16815d1d0 0x7fb1681f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.089+0000 7fb1677fe640 1 -- 172.21.15.124:0/2568639369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb160009b30 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb1677fe640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb15800d5f0 tx=0x7fb15800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1580125f0 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb158012750 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1681f6c20 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb158012b00 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.090+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb168160820 con 0x7fb1680ad640 2023-12-13T23:57:03.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.091+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16815a970 con 0x7fb1680ad640 2023-12-13T23:57:03.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.091+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb158012c60 con 0x7fb1680ad640 2023-12-13T23:57:03.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.092+0000 7fb14ffff640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 0x7fb15406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.092+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb15808a080 con 0x7fb1680ad640 2023-12-13T23:57:03.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.092+0000 7fb166ffd640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 0x7fb15406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.092 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.092+0000 7fb166ffd640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 0x7fb15406d680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb15c003cc0 tx=0x7fb15c0079b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.096+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb16815a970 con 0x7fb1680ad640 2023-12-13T23:57:03.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.261+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb16815a970 con 0x7fb1680ad640 2023-12-13T23:57:03.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.262+0000 7fb14ffff640 1 -- 172.21.15.124:0/2568639369 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v38) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb16815a970 con 0x7fb1680ad640 2023-12-13T23:57:03.261 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 msgr2=0x7fb15406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 0x7fb15406d680 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb15c003cc0 tx=0x7fb15c0079b0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 msgr2=0x7fb1681f5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fb15800d5f0 tx=0x7fb15800dab0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 shutdown_connections 2023-12-13T23:57:03.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb15406b1c0 0x7fb15406d680 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb16815d1d0 0x7fb1681f22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb1680aeb80 0x7fb1681602e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 --2- 172.21.15.124:0/2568639369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1680ad640 0x7fb1681f5fd0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 >> 172.21.15.124:0/2568639369 conn(0x7fb168070e70 msgr2=0x7fb1680af780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 shutdown_connections 2023-12-13T23:57:03.264 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.264+0000 7fb16dfe5640 1 -- 172.21.15.124:0/2568639369 wait complete. 2023-12-13T23:57:03.274 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:03.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.377+0000 7fba314cf640 1 Processor -- start 2023-12-13T23:57:03.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.377+0000 7fba314cf640 1 -- start start 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba314cf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba2c15d270 0x7fba2c06ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba314cf640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0d0c40 con 0x7fba2c0709c0 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba314cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2c0ced90 con 0x7fba2c0cdee0 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba314cf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba2c0cf070 con 0x7fba2c15d270 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba2a7fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fba2c0709c0 0x7fba2c0cd7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41976/0 (socket says 172.21.15.124:41976) 2023-12-13T23:57:03.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.378+0000 7fba2a7fc640 1 -- 172.21.15.124:0/2294130358 learned_addr learned my addr 172.21.15.124:0/2294130358 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899505580 0 0) 0x7fba2c0d0c40 con 0x7fba2c0709c0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14003520 con 0x7fba2c0709c0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2affd640 1 --2- 172.21.15.124:0/2294130358 >> v2:172.21.15.203:3300/0 conn(0x7fba2c15d270 0x7fba2c06ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2081513987 0 0) 0x7fba14003520 con 0x7fba2c0709c0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 >> v2:172.21.15.203:3300/0 conn(0x7fba2c15d270 msgr2=0x7fba2c06ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 --2- 172.21.15.124:0/2294130358 >> v2:172.21.15.203:3300/0 conn(0x7fba2c15d270 0x7fba2c06ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 >> v1:172.21.15.203:6789/0 conn(0x7fba2c0cdee0 legacy=0x7fba2c0cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c0d08f0 con 0x7fba2c0709c0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fba20001670 con 0x7fba2c0709c0 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2affd640 1 --2- 172.21.15.124:0/2294130358 >> v2:172.21.15.203:3300/0 conn(0x7fba2c15d270 0x7fba2c06ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 >> v1:172.21.15.124:6789/0 conn(0x7fba2c0709c0 legacy=0x7fba2c0cd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba14003970 0x7fba14003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba140047c0 0x7fba14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 0x7fba1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c0cf070 con 0x7fba140052a0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1400c170 con 0x7fba14003970 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1400c330 con 0x7fba140047c0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 0x7fba1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2affd640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba14003970 0x7fba14003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2a7fc640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba140047c0 0x7fba14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 -- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba140047c0 msgr2=0x7fba14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba140047c0 0x7fba14004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 -- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba14003970 msgr2=0x7fba14003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba14003970 0x7fba14003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2b7fe640 1 -- 172.21.15.124:0/2294130358 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba14003520 con 0x7fba140052a0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.379+0000 7fba2a7fc640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba140047c0 0x7fba14004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.380+0000 7fba2b7fe640 1 -- 172.21.15.124:0/2294130358 reap_dead start 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.380+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 0x7fba1400b980 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0b2bc0 tx=0x7fba140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a98ca63dfbe1707 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.380+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240052d0 con 0x7fba140052a0 2023-12-13T23:57:03.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.380+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2402e050 con 0x7fba140052a0 2023-12-13T23:57:03.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.380+0000 7fba29ffb640 1 -- 172.21.15.124:0/2294130358 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba240035f0 con 0x7fba140052a0 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 -- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 msgr2=0x7fba1400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 0x7fba1400b980 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0b2bc0 tx=0x7fba140057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 -- 172.21.15.124:0/2294130358 shutdown_connections 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 --2- 172.21.15.124:0/2294130358 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba140052a0 0x7fba1400b980 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 -- 172.21.15.124:0/2294130358 >> 172.21.15.124:0/2294130358 conn(0x7fba2c0b2c10 msgr2=0x7fba2c0b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 -- 172.21.15.124:0/2294130358 shutdown_connections 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.381+0000 7fba314cf640 1 -- 172.21.15.124:0/2294130358 wait complete. 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.382+0000 7fba314cf640 1 Processor -- start 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.382+0000 7fba314cf640 1 -- start start 2023-12-13T23:57:03.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba140052a0 0x7fba2c1f3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba2c15a650 0x7fba2c15eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c15e640 con 0x7fba2c15f420 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c15f8a0 con 0x7fba140052a0 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2c1f4500 con 0x7fba2c15a650 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57872/0 (socket says 172.21.15.124:57872) 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 -- 172.21.15.124:0/101320346 learned_addr learned my addr 172.21.15.124:0/101320346 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2affd640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba140052a0 0x7fba2c1f3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2a7fc640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba2c15a650 0x7fba2c15eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 -- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba2c15a650 msgr2=0x7fba2c15eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba2c15a650 0x7fba2c15eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 -- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba140052a0 msgr2=0x7fba2c1f3fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba140052a0 0x7fba2c1f3fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 -- 172.21.15.124:0/101320346 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c1610f0 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba2b7fe640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fba2402bc80 tx=0x7fba24001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2403d640 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2403d7a0 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba2c0c4890 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2403db30 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.383+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba2c0c4e50 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.384+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2c067fc0 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.385+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fba2403dc90 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.385+0000 7fba137fe640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 0x7fba1806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.385+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fba240aa860 con 0x7fba2c15f420 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.386+0000 7fba2affd640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 0x7fba1806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.386+0000 7fba2affd640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 0x7fba1806d680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba1c007380 tx=0x7fba1c00b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.389+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fba24075490 con 0x7fba2c15f420 2023-12-13T23:57:03.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.556+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fba2c067420 con 0x7fba2c15f420 2023-12-13T23:57:03.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.557+0000 7fba137fe640 1 -- 172.21.15.124:0/101320346 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v38) v1 ==== 100+0+11 (secure 0 0 0) 0x7fba24007a30 con 0x7fba2c15f420 2023-12-13T23:57:03.556 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:03.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.558+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 msgr2=0x7fba1806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.558+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 0x7fba1806d680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fba1c007380 tx=0x7fba1c00b040 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 msgr2=0x7fba2c15d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fba2402bc80 tx=0x7fba24001120 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 shutdown_connections 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba1806b1c0 0x7fba1806d680 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba2c15f420 0x7fba2c15d8f0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba2c15a650 0x7fba2c15eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 --2- 172.21.15.124:0/101320346 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba140052a0 0x7fba2c1f3fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 >> 172.21.15.124:0/101320346 conn(0x7fba2c0b2c10 msgr2=0x7fba2c0ca120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 shutdown_connections 2023-12-13T23:57:03.561 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.559+0000 7fba314cf640 1 -- 172.21.15.124:0/101320346 wait complete. 2023-12-13T23:57:03.569 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-12-13T23:57:03.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.671+0000 7f29044e0640 1 Processor -- start 2023-12-13T23:57:03.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.671+0000 7f29044e0640 1 -- start start 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.672+0000 7f29044e0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28fc15d330 0x7f28fc0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.672+0000 7f29044e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 0x7f28fc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f29044e0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28fc0b2b10 con 0x7f28fc0d2910 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f29044e0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28fc0d1590 con 0x7f28fc15d330 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f29044e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc0d1870 con 0x7f28fc0d3900 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f2902a7d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 0x7f28fc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28fc15d330 0x7f28fc0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f2902a7d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 0x7f28fc0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:36376/0 (socket says 172.21.15.124:36376) 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f2901a7b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f28fc0d2910 0x7f28fc0d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:45966/0 (socket says 172.21.15.124:45966) 2023-12-13T23:57:03.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f28fc15d330 0x7f28fc0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:54990/0 (socket says 172.21.15.124:54990) 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f2902a7d640 1 -- 172.21.15.124:0/2246982015 learned_addr learned my addr 172.21.15.124:0/2246982015 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 -- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 msgr2=0x7f28fc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 --2- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 0x7f28fc0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 -- 172.21.15.124:0/2246982015 >> v1:172.21.15.203:6789/0 conn(0x7f28fc0d2910 legacy=0x7f28fc0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290227c640 1 -- 172.21.15.124:0/2246982015 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28fc0d1b50 con 0x7f28fc15d330 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.673+0000 7f290127a640 1 -- 172.21.15.124:0/2246982015 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058162403 0 0) 0x7f28fc0b2b10 con 0x7f28fc0d2910 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.674+0000 7f290227c640 1 --2- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc15d330 0x7f28fc0b8b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28f80097a0 tx=0x7f28f802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65fdb6d0cfe53aa1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.674+0000 7f290127a640 1 -- 172.21.15.124:0/2246982015 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f802e070 con 0x7f28fc15d330 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.674+0000 7f290127a640 1 -- 172.21.15.124:0/2246982015 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f80315a0 con 0x7f28fc15d330 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.674+0000 7f290127a640 1 -- 172.21.15.124:0/2246982015 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28f8031930 con 0x7f28fc15d330 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 -- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc15d330 msgr2=0x7f28fc0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 --2- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc15d330 0x7f28fc0b8b80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28f80097a0 tx=0x7f28f802b840 comp rx=0 tx=0).stop 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 -- 172.21.15.124:0/2246982015 shutdown_connections 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 --2- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc0d3900 0x7f28fc0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.675 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 --2- 172.21.15.124:0/2246982015 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc15d330 0x7f28fc0b8b80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 -- 172.21.15.124:0/2246982015 >> 172.21.15.124:0/2246982015 conn(0x7f28fc072020 msgr2=0x7f28fc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 -- 172.21.15.124:0/2246982015 shutdown_connections 2023-12-13T23:57:03.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.675+0000 7f29044e0640 1 -- 172.21.15.124:0/2246982015 wait complete. 2023-12-13T23:57:03.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.676+0000 7f29044e0640 1 Processor -- start 2023-12-13T23:57:03.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.676+0000 7f29044e0640 1 -- start start 2023-12-13T23:57:03.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc0d2910 0x7f28fc14b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 0x7f28fc0b8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc15f260 con 0x7f28fc0d3900 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc15e650 con 0x7f28fc0d2910 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28fc15e930 con 0x7f28fc15d330 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57886/0 (socket says 172.21.15.124:57886) 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 -- 172.21.15.124:0/2311637636 learned_addr learned my addr 172.21.15.124:0/2311637636 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f290227c640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc0d2910 0x7f28fc14b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2902a7d640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 0x7f28fc0b8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 -- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 msgr2=0x7f28fc0b8bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 0x7f28fc0b8bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 -- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc0d2910 msgr2=0x7f28fc14b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc0d2910 0x7f28fc14b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 -- 172.21.15.124:0/2311637636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f8009400 con 0x7f28fc0d3900 2023-12-13T23:57:03.679 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2902a7d640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 0x7f28fc0b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f2901a7b640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c430 tx=0x7f28ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec0074e0 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ec007660 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28fc15ec10 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.677+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28ec004a20 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.678+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28fc15d920 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.679+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28fc0d2dc0 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.679+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f28ec004b80 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.679+0000 7f28f2ffd640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 0x7f28e006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.679+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f28ec089960 con 0x7f28fc0d3900 2023-12-13T23:57:03.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.679+0000 7f290227c640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 0x7f28e006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.680+0000 7f290227c640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 0x7f28e006d6b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f28f8036000 tx=0x7f28f803d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.682 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.684+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f28ec054470 con 0x7f28fc0d3900 2023-12-13T23:57:03.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.849+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f28fc069400 con 0x7f28fc0d3900 2023-12-13T23:57:03.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.849+0000 7f28f2ffd640 1 -- 172.21.15.124:0/2311637636 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v38) v1 ==== 97+0+11 (secure 0 0 0) 0x7f28ec057a90 con 0x7f28fc0d3900 2023-12-13T23:57:03.849 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:03.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 msgr2=0x7f28e006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 0x7f28e006d6b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f28f8036000 tx=0x7f28f803d040 comp rx=0 tx=0).stop 2023-12-13T23:57:03.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 msgr2=0x7f28fc0b86b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f28ec00c430 tx=0x7f28ec00c8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 shutdown_connections 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f28e006b1f0 0x7f28e006d6b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28fc15d330 0x7f28fc0b8bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28fc0d3900 0x7f28fc0b86b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 --2- 172.21.15.124:0/2311637636 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f28fc0d2910 0x7f28fc14b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 >> 172.21.15.124:0/2311637636 conn(0x7f28fc072020 msgr2=0x7f28fc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 shutdown_connections 2023-12-13T23:57:03.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.851+0000 7f29044e0640 1 -- 172.21.15.124:0/2311637636 wait complete. 2023-12-13T23:57:03.861 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:03.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.963+0000 7f32b7eda640 1 Processor -- start 2023-12-13T23:57:03.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.963+0000 7f32b7eda640 1 -- start start 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b7eda640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f32b015d2b0 0x7f32b0071c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b7eda640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f32b00ae480 0x7f32b00721c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b7eda640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00ae910 con 0x7f32b0072840 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b7eda640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00b2200 con 0x7f32b015d2b0 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b7eda640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00b24e0 con 0x7f32b00ae480 2023-12-13T23:57:03.963 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f32b015d2b0 0x7f32b0071c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f32b015d2b0 0x7f32b0071c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57892/0 (socket says 172.21.15.124:57892) 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 -- 172.21.15.124:0/4168516554 learned_addr learned my addr 172.21.15.124:0/4168516554 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 -- 172.21.15.124:0/4168516554 >> v2:172.21.15.203:3300/0 conn(0x7f32b00ae480 msgr2=0x7f32b00721c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 --2- 172.21.15.124:0/4168516554 >> v2:172.21.15.203:3300/0 conn(0x7f32b00ae480 0x7f32b00721c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 -- 172.21.15.124:0/4168516554 >> v1:172.21.15.203:6789/0 conn(0x7f32b0072840 legacy=0x7f32b006d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:03.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.964+0000 7f32b5c76640 1 -- 172.21.15.124:0/4168516554 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00b40a0 con 0x7f32b015d2b0 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b5c76640 1 --2- 172.21.15.124:0/4168516554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b015d2b0 0x7f32b0071c80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f32b0160e00 tx=0x7f32ac02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c839e57e012f2e4b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b4c74640 1 -- 172.21.15.124:0/4168516554 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0317e0 con 0x7f32b015d2b0 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b4c74640 1 -- 172.21.15.124:0/4168516554 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac031940 con 0x7f32b015d2b0 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b4c74640 1 -- 172.21.15.124:0/4168516554 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac031cd0 con 0x7f32b015d2b0 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b7eda640 1 -- 172.21.15.124:0/4168516554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b015d2b0 msgr2=0x7f32b0071c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.965 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.965+0000 7f32b7eda640 1 --2- 172.21.15.124:0/4168516554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b015d2b0 0x7f32b0071c80 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f32b0160e00 tx=0x7f32ac02b880 comp rx=0 tx=0).stop 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 -- 172.21.15.124:0/4168516554 shutdown_connections 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 --2- 172.21.15.124:0/4168516554 >> v2:172.21.15.203:3300/0 conn(0x7f32b00ae480 0x7f32b00721c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 --2- 172.21.15.124:0/4168516554 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b015d2b0 0x7f32b0071c80 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 -- 172.21.15.124:0/4168516554 >> 172.21.15.124:0/4168516554 conn(0x7f32b00b4720 msgr2=0x7f32b00b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 -- 172.21.15.124:0/4168516554 shutdown_connections 2023-12-13T23:57:03.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.966+0000 7f32b7eda640 1 -- 172.21.15.124:0/4168516554 wait complete. 2023-12-13T23:57:03.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 Processor -- start 2023-12-13T23:57:03.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 -- start start 2023-12-13T23:57:03.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32b015d2b0 0x7f32b01580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.968 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 0x7f32b0152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00aef00 con 0x7f32b0072840 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0156aa0 con 0x7f32b0149fc0 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.967+0000 7f32b7eda640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0156d80 con 0x7f32b015d2b0 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57908/0 (socket says 172.21.15.124:57908) 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 -- 172.21.15.124:0/1226776605 learned_addr learned my addr 172.21.15.124:0/1226776605 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b6477640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 0x7f32b0152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5475640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32b015d2b0 0x7f32b01580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 -- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32b015d2b0 msgr2=0x7f32b01580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32b015d2b0 0x7f32b01580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 -- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 msgr2=0x7f32b0152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 0x7f32b0152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:03.969 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 -- 172.21.15.124:0/1226776605 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac0094d0 con 0x7f32b0072840 2023-12-13T23:57:03.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b6477640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 0x7f32b0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:03.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b5c76640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f32ac000c00 tx=0x7f32ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.972 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac032880 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32ac0329e0 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32ac032d90 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b00b2af0 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.968+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b0157340 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.969+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b0069400 con 0x7f32b0072840 2023-12-13T23:57:03.973 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.970+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f32ac04d3f0 con 0x7f32b0072840 2023-12-13T23:57:03.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.970+0000 7f32a67fc640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 0x7f329406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:03.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.970+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f32ac0aba70 con 0x7f32b0072840 2023-12-13T23:57:03.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.970+0000 7f32b5475640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 0x7f329406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:03.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.971+0000 7f32b5475640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 0x7f329406d6b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f32b015c070 tx=0x7f32a0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:03.974 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:03.974+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f32ac076610 con 0x7f32b0072840 2023-12-13T23:57:04.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.175+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b00cfad0 con 0x7f32b0072840 2023-12-13T23:57:04.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.176+0000 7f32a67fc640 1 -- 172.21.15.124:0/1226776605 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4923 (secure 0 0 0) 0x7f32ac079c30 con 0x7f32b0072840 2023-12-13T23:57:04.177 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:04.177 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:26.366668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 15 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 msgr2=0x7f329406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 0x7f329406d6b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f32b015c070 tx=0x7f32a0008040 comp rx=0 tx=0).stop 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 msgr2=0x7f32b01564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f32ac000c00 tx=0x7f32ac0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 shutdown_connections 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f329406b1f0 0x7f329406d6b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f32b0149fc0 0x7f32b0152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f32b015d2b0 0x7f32b01580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 --2- 172.21.15.124:0/1226776605 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f32b0072840 0x7f32b01564c0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 >> 172.21.15.124:0/1226776605 conn(0x7f32b00b4720 msgr2=0x7f32b015e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 shutdown_connections 2023-12-13T23:57:04.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.178+0000 7f32b7eda640 1 -- 172.21.15.124:0/1226776605 wait complete. 2023-12-13T23:57:04.188 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:04.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.288+0000 7ff1bb989640 1 Processor -- start 2023-12-13T23:57:04.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.288+0000 7ff1bb989640 1 -- start start 2023-12-13T23:57:04.288 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1bb989640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff1b40ceb70 0x7ff1b40cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1bb989640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b40b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1bb989640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1b4071770 con 0x7ff1b4144980 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1bb989640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1b415ced0 con 0x7ff1b40ceb70 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1bb989640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b40b4ce0 con 0x7ff1b40b4370 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff1b40ceb70 0x7ff1b40cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff1b40ceb70 0x7ff1b40cc450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57922/0 (socket says 172.21.15.124:57922) 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 -- 172.21.15.124:0/999644165 learned_addr learned my addr 172.21.15.124:0/999644165 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 -- 172.21.15.124:0/999644165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 msgr2=0x7ff1b40b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 --2- 172.21.15.124:0/999644165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b40b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 -- 172.21.15.124:0/999644165 >> v1:172.21.15.203:6789/0 conn(0x7ff1b4144980 legacy=0x7ff1b415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.289+0000 7ff1b8f24640 1 -- 172.21.15.124:0/999644165 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1b40ccdc0 con 0x7ff1b40ceb70 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.290+0000 7ff1b8f24640 1 --2- 172.21.15.124:0/999644165 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40cc450 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009870 tx=0x7ff1a402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a457fb04872805 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.290+0000 7ff1abfff640 1 -- 172.21.15.124:0/999644165 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4031640 con 0x7ff1b40ceb70 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.290+0000 7ff1abfff640 1 -- 172.21.15.124:0/999644165 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1a40317a0 con 0x7ff1b40ceb70 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.290+0000 7ff1abfff640 1 -- 172.21.15.124:0/999644165 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1a4031b30 con 0x7ff1b40ceb70 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 -- 172.21.15.124:0/999644165 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b40ceb70 msgr2=0x7ff1b40cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 --2- 172.21.15.124:0/999644165 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40cc450 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ff1a4009870 tx=0x7ff1a402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 -- 172.21.15.124:0/999644165 shutdown_connections 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 --2- 172.21.15.124:0/999644165 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b40b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 --2- 172.21.15.124:0/999644165 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40cc450 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 -- 172.21.15.124:0/999644165 >> 172.21.15.124:0/999644165 conn(0x7ff1b406a4d0 msgr2=0x7ff1b406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 -- 172.21.15.124:0/999644165 shutdown_connections 2023-12-13T23:57:04.291 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.291+0000 7ff1bb989640 1 -- 172.21.15.124:0/999644165 wait complete. 2023-12-13T23:57:04.293 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.292+0000 7ff1bb989640 1 Processor -- start 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.292+0000 7ff1bb989640 1 -- start start 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b4158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b40c9fa0 con 0x7ff1b4144980 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b40ca280 con 0x7ff1b40ceb70 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1bb989640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1b40ca560 con 0x7ff1b40b4370 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57926/0 (socket says 172.21.15.124:57926) 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 -- 172.21.15.124:0/3381775770 learned_addr learned my addr 172.21.15.124:0/3381775770 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 -- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 msgr2=0x7ff1b4158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b4158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 -- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b40ceb70 msgr2=0x7ff1b40c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.294 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 -- 172.21.15.124:0/3381775770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a40094d0 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.293+0000 7ff1b9f26640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009fd0 tx=0x7ff1ac00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.294+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac011830 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.294+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1b40ca840 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.294+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1ac011990 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.294+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1ac011d20 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.294+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1b40c9c90 con 0x7ff1b4144980 2023-12-13T23:57:04.295 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.295+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1b40cad60 con 0x7ff1b4144980 2023-12-13T23:57:04.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.295+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff1ac01b460 con 0x7ff1b4144980 2023-12-13T23:57:04.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.295+0000 7ff1a9ffb640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 0x7ff19806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.295+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7ff1ac089a40 con 0x7ff1b4144980 2023-12-13T23:57:04.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.296+0000 7ff1b9725640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 0x7ff19806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.296 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.296+0000 7ff1b9725640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 0x7ff19806d680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000ab90 tx=0x7ff1b0009230 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.298 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.300+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff1b40cad60 con 0x7ff1b4144980 2023-12-13T23:57:04.466 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.467+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1b40cad60 con 0x7ff1b4144980 2023-12-13T23:57:04.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.468+0000 7ff1a9ffb640 1 -- 172.21.15.124:0/3381775770 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7ff1b40cad60 con 0x7ff1b4144980 2023-12-13T23:57:04.468 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:04.468 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:04.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 msgr2=0x7ff19806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 0x7ff19806d680 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000ab90 tx=0x7ff1b0009230 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 msgr2=0x7ff1b41546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ff1ac009fd0 tx=0x7ff1ac00b8e0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 shutdown_connections 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff19806b1c0 0x7ff19806d680 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff1b40ceb70 0x7ff1b40c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1b4144980 0x7ff1b41546b0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 --2- 172.21.15.124:0/3381775770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff1b40b4370 0x7ff1b4158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 >> 172.21.15.124:0/3381775770 conn(0x7ff1b406a4d0 msgr2=0x7ff1b4071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 shutdown_connections 2023-12-13T23:57:04.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.470+0000 7ff1bb989640 1 -- 172.21.15.124:0/3381775770 wait complete. 2023-12-13T23:57:04.480 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-13T23:57:04.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:04.579 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.580+0000 7faeabb0c640 1 Processor -- start 2023-12-13T23:57:04.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 -- start start 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faea40d2910 0x7faea40d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faea40d3730 0x7faea40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea40b2b10 con 0x7faea415d330 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea40d3bc0 con 0x7faea40d3730 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeabb0c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea40d1590 con 0x7faea40d2910 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.581+0000 7faeaa0a9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faea40d3730 0x7faea40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faea40d3730 0x7faea40b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57938/0 (socket says 172.21.15.124:57938) 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 -- 172.21.15.124:0/1477438790 learned_addr learned my addr 172.21.15.124:0/1477438790 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 -- 172.21.15.124:0/1477438790 >> v2:172.21.15.203:3300/0 conn(0x7faea40d2910 msgr2=0x7faea40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 --2- 172.21.15.124:0/1477438790 >> v2:172.21.15.203:3300/0 conn(0x7faea40d2910 0x7faea40d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 -- 172.21.15.124:0/1477438790 >> v1:172.21.15.203:6789/0 conn(0x7faea415d330 legacy=0x7faea40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 -- 172.21.15.124:0/1477438790 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea40d1870 con 0x7faea40d3730 2023-12-13T23:57:04.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.582+0000 7faeaa0a9640 1 --2- 172.21.15.124:0/1477438790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea40b6390 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7faea40d1540 tx=0x7fae9c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58df97aa1b69454 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faea88a6640 1 -- 172.21.15.124:0/1477438790 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c0317e0 con 0x7faea40d3730 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faea88a6640 1 -- 172.21.15.124:0/1477438790 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c031940 con 0x7faea40d3730 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faea88a6640 1 -- 172.21.15.124:0/1477438790 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c031cd0 con 0x7faea40d3730 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 -- 172.21.15.124:0/1477438790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 msgr2=0x7faea40b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 --2- 172.21.15.124:0/1477438790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea40b6390 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7faea40d1540 tx=0x7fae9c02b880 comp rx=0 tx=0).stop 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 -- 172.21.15.124:0/1477438790 shutdown_connections 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 --2- 172.21.15.124:0/1477438790 >> v2:172.21.15.203:3300/0 conn(0x7faea40d2910 0x7faea40d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 --2- 172.21.15.124:0/1477438790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea40b6390 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 -- 172.21.15.124:0/1477438790 >> 172.21.15.124:0/1477438790 conn(0x7faea4072020 msgr2=0x7faea4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 -- 172.21.15.124:0/1477438790 shutdown_connections 2023-12-13T23:57:04.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.583+0000 7faeabb0c640 1 -- 172.21.15.124:0/1477438790 wait complete. 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.584+0000 7faeabb0c640 1 Processor -- start 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.584+0000 7faeabb0c640 1 -- start start 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faea40d2910 0x7faea4154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea415d330 0x7faea40c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c9b90 con 0x7faea40d3730 2023-12-13T23:57:04.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c7290 con 0x7faea415d330 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeabb0c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea415f8a0 con 0x7faea40d2910 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faea90a7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faea90a7640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57948/0 (socket says 172.21.15.124:57948) 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faea90a7640 1 -- 172.21.15.124:0/3596120431 learned_addr learned my addr 172.21.15.124:0/3596120431 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faeaa0a9640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea415d330 0x7faea40c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.585+0000 7faea98a8640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faea40d2910 0x7faea4154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 -- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faea40d2910 msgr2=0x7faea4154aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faea40d2910 0x7faea4154aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 -- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea415d330 msgr2=0x7faea40c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea415d330 0x7faea40c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 -- 172.21.15.124:0/3596120431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0094d0 con 0x7faea40d3730 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faea90a7640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fae9400b4a0 tx=0x7fae9400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94010610 con 0x7faea40d3730 2023-12-13T23:57:04.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae94010770 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae94010b20 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea415fe20 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.586+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea40c6c40 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.587+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea40c7400 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.587+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fae94010c80 con 0x7faea40d3730 2023-12-13T23:57:04.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.588+0000 7fae9a7fc640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 0x7fae8806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.588+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fae94089710 con 0x7faea40d3730 2023-12-13T23:57:04.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.588+0000 7faea98a8640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 0x7fae8806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.588+0000 7faea98a8640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 0x7fae8806d680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faea0009f60 tx=0x7faea00094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.592+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fae94054340 con 0x7faea40d3730 2023-12-13T23:57:04.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.792+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea4069400 con 0x7faea40d3730 2023-12-13T23:57:04.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.793+0000 7fae9a7fc640 1 -- 172.21.15.124:0/3596120431 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4923 (secure 0 0 0) 0x7fae94057960 con 0x7faea40d3730 2023-12-13T23:57:04.793 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:04.794 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:26.366668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 15 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 msgr2=0x7fae8806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 0x7fae8806d680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7faea0009f60 tx=0x7faea00094f0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 msgr2=0x7faea415f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fae9400b4a0 tx=0x7fae9400b960 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 shutdown_connections 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fae8806b1c0 0x7fae8806d680 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faea415d330 0x7faea40c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faea40d3730 0x7faea415f360 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 --2- 172.21.15.124:0/3596120431 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faea40d2910 0x7faea4154aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 >> 172.21.15.124:0/3596120431 conn(0x7faea4072020 msgr2=0x7faea4072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 shutdown_connections 2023-12-13T23:57:04.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.795+0000 7faeabb0c640 1 -- 172.21.15.124:0/3596120431 wait complete. 2023-12-13T23:57:04.805 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:04.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.908+0000 7fb9f09c9640 1 Processor -- start 2023-12-13T23:57:04.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.908+0000 7fb9f09c9640 1 -- start start 2023-12-13T23:57:04.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9f09c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb9ec144980 0x7fb9ec15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9f09c9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec071770 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9f09c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9ec0ccb60 con 0x7fb9ec0b4370 2023-12-13T23:57:04.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9f09c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9ec0ccdc0 con 0x7fb9ec144980 2023-12-13T23:57:04.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb9ec0ceb70 0x7fb9ec0cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41982/0 (socket says 172.21.15.124:41982) 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e3fff640 1 -- 172.21.15.124:0/1471612353 learned_addr learned my addr 172.21.15.124:0/1471612353 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273969055 0 0) 0x7fb9ec071770 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d8003520 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9ea59c640 1 --2- 172.21.15.124:0/1471612353 >> v2:172.21.15.203:3300/0 conn(0x7fb9ec144980 0x7fb9ec15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4100613665 0 0) 0x7fb9d8003520 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 >> v2:172.21.15.203:3300/0 conn(0x7fb9ec144980 msgr2=0x7fb9ec15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 --2- 172.21.15.124:0/1471612353 >> v2:172.21.15.203:3300/0 conn(0x7fb9ec144980 0x7fb9ec15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 >> v1:172.21.15.203:6789/0 conn(0x7fb9ec0b4370 legacy=0x7fb9ec071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0cd0a0 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fb9d4001670 con 0x7fb9ec0ceb70 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 >> v1:172.21.15.124:6789/0 conn(0x7fb9ec0ceb70 legacy=0x7fb9ec0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9d8003970 0x7fb9d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9d80047c0 0x7fb9d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0ccdc0 con 0x7fb9d8005180 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d800c020 con 0x7fb9d8003970 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.909+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9d800c1e0 con 0x7fb9d80047c0 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ea59c640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9d8003970 0x7fb9d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9e3fff640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9d80047c0 0x7fb9d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 -- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9d80047c0 msgr2=0x7fb9d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9d80047c0 0x7fb9d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 -- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9d8003970 msgr2=0x7fb9d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9d8003970 0x7fb9d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 -- 172.21.15.124:0/1471612353 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8003520 con 0x7fb9d8005180 2023-12-13T23:57:04.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.910+0000 7fb9ead9d640 1 -- 172.21.15.124:0/1471612353 reap_dead start 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9d800b980 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001e50 tx=0x7fb9d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95071ef6b8e9bccf server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4003b70 con 0x7fb9d8005180 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9e4003d10 con 0x7fb9d8005180 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9e9d9b640 1 -- 172.21.15.124:0/1471612353 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e4005310 con 0x7fb9d8005180 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 -- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 msgr2=0x7fb9d800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9d800b980 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fb9e4001e50 tx=0x7fb9d80057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 -- 172.21.15.124:0/1471612353 shutdown_connections 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/1471612353 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9d800b980 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 -- 172.21.15.124:0/1471612353 >> 172.21.15.124:0/1471612353 conn(0x7fb9ec06a4d0 msgr2=0x7fb9ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 -- 172.21.15.124:0/1471612353 shutdown_connections 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.911+0000 7fb9f09c9640 1 -- 172.21.15.124:0/1471612353 wait complete. 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 Processor -- start 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 -- start start 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 0x7fb9ec13b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 0x7fb9ec0b3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0b3bd0 con 0x7fb9d8005180 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0ad130 con 0x7fb9ec15b5f0 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9f09c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9ec0ad410 con 0x7fb9ec0b31e0 2023-12-13T23:57:04.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9ea59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57962/0 (socket says 172.21.15.124:57962) 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.913+0000 7fb9ea59c640 1 -- 172.21.15.124:0/3342088996 learned_addr learned my addr 172.21.15.124:0/3342088996 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 0x7fb9ec0b3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9e3fff640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 0x7fb9ec13b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 -- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 msgr2=0x7fb9ec0b3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 0x7fb9ec0b3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 -- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 msgr2=0x7fb9ec13b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 0x7fb9ec13b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 -- 172.21.15.124:0/3342088996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec0ad6f0 con 0x7fb9d8005180 2023-12-13T23:57:04.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ead9d640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 0x7fb9ec0b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9e3fff640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 0x7fb9ec13b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9ea59c640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0118e0 tx=0x7fb9dc011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc00faf0 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc00fc50 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9ec0ad9d0 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc014440 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.914+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9ec0ae1d0 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.915+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9ec069400 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.916+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb9dc01d400 con 0x7fb9d8005180 2023-12-13T23:57:04.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.916+0000 7fb9e2ffd640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 0x7fb9cc06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:04.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.916+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7fb9dc08a460 con 0x7fb9d8005180 2023-12-13T23:57:04.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.916+0000 7fb9e3fff640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 0x7fb9cc06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:04.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.917+0000 7fb9e3fff640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 0x7fb9cc06d6b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec15c1b0 tx=0x7fb9d4009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:04.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:04.920+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb9dc055000 con 0x7fb9d8005180 2023-12-13T23:57:05.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.124+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9ec15d450 con 0x7fb9d8005180 2023-12-13T23:57:05.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.125+0000 7fb9e2ffd640 1 -- 172.21.15.124:0/3342088996 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4923 (secure 0 0 0) 0x7fb9dc058620 con 0x7fb9d8005180 2023-12-13T23:57:05.124 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:05.124 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:56:18.320734+0000","modified":"2023-12-13T23:56:26.366668+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5271},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5271":{"gid":5271,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/4122494545","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":4122494545},{"type":"v1","addr":"172.21.15.124:6835","nonce":4122494545}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-12-13T23:57:05.124 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 15 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.126+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 msgr2=0x7fb9cc06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.126+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 0x7fb9cc06d6b0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec15c1b0 tx=0x7fb9d4009040 comp rx=0 tx=0).stop 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.126+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 msgr2=0x7fb9ec0ac8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.126+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0118e0 tx=0x7fb9dc011da0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 shutdown_connections 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb9cc06b1f0 0x7fb9cc06d6b0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb9ec15b5f0 0x7fb9ec13b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb9ec0b31e0 0x7fb9ec0b3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 --2- 172.21.15.124:0/3342088996 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb9d8005180 0x7fb9ec0ac8e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 >> 172.21.15.124:0/3342088996 conn(0x7fb9ec06a4d0 msgr2=0x7fb9ec143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 shutdown_connections 2023-12-13T23:57:05.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.127+0000 7fb9f09c9640 1 -- 172.21.15.124:0/3342088996 wait complete. 2023-12-13T23:57:05.137 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:05.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 Processor -- start 2023-12-13T23:57:05.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 -- start start 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f64680b1400 0x7f646815a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f646815e730 0x7f646815aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f646815e2a0 con 0x7f6468143490 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f646815b520 con 0x7f64680b1400 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.238+0000 7f646dc3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6468143870 con 0x7f646815e730 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f64680b1400 0x7f646815a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f64677fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6468143490 0x7f64680b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41992/0 (socket says 172.21.15.124:41992) 2023-12-13T23:57:05.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f64680b1400 0x7f646815a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57976/0 (socket says 172.21.15.124:57976) 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f64677fe640 1 -- 172.21.15.124:0/834539476 learned_addr learned my addr 172.21.15.124:0/834539476 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f6467fff640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f646815e730 0x7f646815aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 -- 172.21.15.124:0/834539476 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f646815e730 msgr2=0x7f646815aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f646815e730 0x7f646815aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 -- 172.21.15.124:0/834539476 >> v1:172.21.15.124:6789/0 conn(0x7f6468143490 legacy=0x7f64680b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f6466ffd640 1 -- 172.21.15.124:0/834539476 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832128221 0 0) 0x7f646815e2a0 con 0x7f6468143490 2023-12-13T23:57:05.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.239+0000 7f645effd640 1 -- 172.21.15.124:0/834539476 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646815b240 con 0x7f64680b1400 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f645effd640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f64680b1400 0x7f646815a4e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f64500097a0 tx=0x7f645002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25308a749719f26 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f6466ffd640 1 -- 172.21.15.124:0/834539476 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450031830 con 0x7f64680b1400 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f6466ffd640 1 -- 172.21.15.124:0/834539476 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6450031990 con 0x7f64680b1400 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f6466ffd640 1 -- 172.21.15.124:0/834539476 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6450031d20 con 0x7f64680b1400 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 -- 172.21.15.124:0/834539476 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f64680b1400 msgr2=0x7f646815a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f64680b1400 0x7f646815a4e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f64500097a0 tx=0x7f645002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 -- 172.21.15.124:0/834539476 shutdown_connections 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f646815e730 0x7f646815aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 --2- 172.21.15.124:0/834539476 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f64680b1400 0x7f646815a4e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 -- 172.21.15.124:0/834539476 >> 172.21.15.124:0/834539476 conn(0x7f646806a4d0 msgr2=0x7f646806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.240+0000 7f646dc3d640 1 -- 172.21.15.124:0/834539476 shutdown_connections 2023-12-13T23:57:05.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.241+0000 7f646dc3d640 1 -- 172.21.15.124:0/834539476 wait complete. 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.242+0000 7f646dc3d640 1 Processor -- start 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.242+0000 7f646dc3d640 1 -- start start 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 0x7f64680adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f646815e730 0x7f64680b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.243 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64680ae200 con 0x7f6468143490 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64681394e0 con 0x7f646815e730 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64681397c0 con 0x7f64680b1400 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57982/0 (socket says 172.21.15.124:57982) 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 -- 172.21.15.124:0/1915230889 learned_addr learned my addr 172.21.15.124:0/1915230889 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f64677fe640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 0x7f64680adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f6467fff640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f646815e730 0x7f64680b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 -- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 msgr2=0x7f64680adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 0x7f64680adcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 -- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f646815e730 msgr2=0x7f64680b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f646815e730 0x7f64680b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 -- 172.21.15.124:0/1915230889 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6450009400 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f64677fe640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 0x7f64680adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f645effd640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f645002bee0 tx=0x7f64500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f645003ba70 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f645003bbf0 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6468139aa0 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64500453f0 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.243+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f646813a060 con 0x7f6468143490 2023-12-13T23:57:05.245 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.244+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6468069400 con 0x7f6468143490 2023-12-13T23:57:05.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.245+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f645004e450 con 0x7f6468143490 2023-12-13T23:57:05.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.245+0000 7f6464ff9640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 0x7f644806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.245+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f64500abe20 con 0x7f6468143490 2023-12-13T23:57:05.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.245+0000 7f64677fe640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 0x7f644806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.246 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.246+0000 7f64677fe640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 0x7f644806d680 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f60 tx=0x7f6458007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.249+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f64500769c0 con 0x7f6468143490 2023-12-13T23:57:05.416 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.417+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64680ae3a0 con 0x7f6468143490 2023-12-13T23:57:05.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.419+0000 7f6464ff9640 1 -- 172.21.15.124:0/1915230889 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+18409 (secure 0 0 0) 0x7f6450035090 con 0x7f6468143490 2023-12-13T23:57:05.418 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:05.418 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":38,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:56:28.028137+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:56:16.668961+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-12-13T23:56:17.292190+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2023-12-13T23:56:25.305690+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 msgr2=0x7f644806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 0x7f644806d680 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f6458000f60 tx=0x7f6458007c10 comp rx=0 tx=0).stop 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 msgr2=0x7f646813a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f645002bee0 tx=0x7f64500046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 shutdown_connections 2023-12-13T23:57:05.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f644806b1c0 0x7f644806d680 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f646815e730 0x7f64680b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6468143490 0x7f646813a580 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 --2- 172.21.15.124:0/1915230889 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f64680b1400 0x7f64680adcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 >> 172.21.15.124:0/1915230889 conn(0x7f646806a4d0 msgr2=0x7f64680cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 shutdown_connections 2023-12-13T23:57:05.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.421+0000 7f646dc3d640 1 -- 172.21.15.124:0/1915230889 wait complete. 2023-12-13T23:57:05.431 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-13T23:57:05.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.533+0000 7f7b55fec640 1 Processor -- start 2023-12-13T23:57:05.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.533+0000 7f7b55fec640 1 -- start start 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b55fec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b5015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b55fec640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7b5015e730 0x7f7b5015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b55fec640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b5015e2a0 con 0x7f7b50143490 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b55fec640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b5015b520 con 0x7f7b5015e730 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b55fec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b50143870 con 0x7f7b500b1400 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4f7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f7b50143490 0x7f7b500b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42008/0 (socket says 172.21.15.124:42008) 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4f7fe640 1 -- 172.21.15.124:0/274033407 learned_addr learned my addr 172.21.15.124:0/274033407 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 863378636 0 0) 0x7f7b5015e2a0 con 0x7f7b50143490 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b38003520 con 0x7f7b50143490 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4effd640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b5015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> v2:172.21.15.203:3300/0 conn(0x7f7b5015e730 0x7f7b5015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2898984636 0 0) 0x7f7b38003520 con 0x7f7b50143490 2023-12-13T23:57:05.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 msgr2=0x7f7b5015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b5015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 >> v2:172.21.15.203:3300/0 conn(0x7f7b5015e730 msgr2=0x7f7b5015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 --2- 172.21.15.124:0/274033407 >> v2:172.21.15.203:3300/0 conn(0x7f7b5015e730 0x7f7b5015aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b5015b240 con 0x7f7b50143490 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f7b40001670 con 0x7f7b50143490 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> v2:172.21.15.203:3300/0 conn(0x7f7b5015e730 0x7f7b5015aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.534+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 >> v1:172.21.15.124:6789/0 conn(0x7f7b50143490 legacy=0x7f7b500b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b38003970 0x7f7b38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b38004d00 0x7f7b38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5015b520 con 0x7f7b38004d00 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b50143870 con 0x7f7b38003970 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3800b980 con 0x7f7b500b1400 2023-12-13T23:57:05.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b38004d00 0x7f7b38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4f7fe640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b38003970 0x7f7b38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4effd640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 -- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 msgr2=0x7f7b380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b500b1400 0x7f7b380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 -- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b38003970 msgr2=0x7f7b38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b38003970 0x7f7b38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.535+0000 7f7b4ffff640 1 -- 172.21.15.124:0/274033407 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b38003520 con 0x7f7b38004d00 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b38004d00 0x7f7b38005180 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009c40 tx=0x7f7b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80edbb62ebc5e2f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48035600 con 0x7f7b38004d00 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b48031050 con 0x7f7b38004d00 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b4e7fc640 1 -- 172.21.15.124:0/274033407 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b48035960 con 0x7f7b38004d00 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 -- 172.21.15.124:0/274033407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b38004d00 msgr2=0x7f7b38005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 --2- 172.21.15.124:0/274033407 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b38004d00 0x7f7b38005180 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f7b48009c40 tx=0x7f7b380057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b4ffff640 1 -- 172.21.15.124:0/274033407 reap_dead start 2023-12-13T23:57:05.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 -- 172.21.15.124:0/274033407 shutdown_connections 2023-12-13T23:57:05.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 -- 172.21.15.124:0/274033407 >> 172.21.15.124:0/274033407 conn(0x7f7b5006a4d0 msgr2=0x7f7b5006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 -- 172.21.15.124:0/274033407 shutdown_connections 2023-12-13T23:57:05.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.536+0000 7f7b55fec640 1 -- 172.21.15.124:0/274033407 wait complete. 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 Processor -- start 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 -- start start 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b5013d510 0x7f7b5015bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b5015bff0 0x7f7b500afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.541 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b5015c470 con 0x7f7b5013cdc0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b500ac240 con 0x7f7b5015bff0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b55fec640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b500aecd0 con 0x7f7b5013d510 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.538+0000 7f7b4f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57992/0 (socket says 172.21.15.124:57992) 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 -- 172.21.15.124:0/891674862 learned_addr learned my addr 172.21.15.124:0/891674862 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 -- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b5013d510 msgr2=0x7f7b5015bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b5013d510 0x7f7b5015bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 -- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b5015bff0 msgr2=0x7f7b500afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4ffff640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b5015bff0 0x7f7b500afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b5015bff0 0x7f7b500afd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 -- 172.21.15.124:0/891674862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b500aefb0 con 0x7f7b5013cdc0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b4f7fe640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7b40004840 tx=0x7f7b40007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000f960 con 0x7f7b5013cdc0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4000fac0 con 0x7f7b5013cdc0 2023-12-13T23:57:05.542 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4000fe70 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b500af290 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.539+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b500af850 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.540+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b50069400 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.540+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7b4001d460 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.541+0000 7f7b37fff640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 0x7f7b3c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.541+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6104+0+0 (secure 0 0 0) 0x7f7b40089e90 con 0x7f7b5013cdc0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.541+0000 7f7b4effd640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 0x7f7b3c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.542+0000 7f7b4effd640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 0x7f7b3c06d680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7b5013da10 tx=0x7f7b44009320 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.545+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7b400584c0 con 0x7f7b5013cdc0 2023-12-13T23:57:05.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.745+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7b500b2820 con 0x7f7b5013cdc0 2023-12-13T23:57:05.744 INFO:tasks.ceph.mon.a.smithi124.stderr:2023-12-13T23:57:05.745+0000 7ffbf0477640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-13T23:57:05.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.747+0000 7f7b37fff640 1 -- 172.21.15.124:0/891674862 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7b40058050 con 0x7f7b5013cdc0 2023-12-13T23:57:05.746 INFO:teuthology.orchestra.run.smithi124.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-13T23:57:05.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 msgr2=0x7f7b3c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 0x7f7b3c06d680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7b5013da10 tx=0x7f7b44009320 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 msgr2=0x7f7b500abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f7b40004840 tx=0x7f7b40007720 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 shutdown_connections 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b3c06b1c0 0x7f7b3c06d680 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7b5015bff0 0x7f7b500afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7b5013d510 0x7f7b5015bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 --2- 172.21.15.124:0/891674862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b5013cdc0 0x7f7b500abd00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 >> 172.21.15.124:0/891674862 conn(0x7f7b5006a4d0 msgr2=0x7f7b50071820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 shutdown_connections 2023-12-13T23:57:05.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.749+0000 7f7b55fec640 1 -- 172.21.15.124:0/891674862 wait complete. 2023-12-13T23:57:05.750 INFO:tasks.ceph.mds.a.smithi124.stderr: -3561> 2023-12-13T23:56:21.958+0000 7f1961a64640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:57:05.760 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-13T23:57:05.778 INFO:tasks.ceph.mds.a.smithi124.stdout:starting mds.a at 2023-12-13T23:57:05.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.864+0000 7f5158b16640 1 Processor -- start 2023-12-13T23:57:05.863 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.864+0000 7f5158b16640 1 -- start start 2023-12-13T23:57:05.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5158b16640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f5154155790 0x7f515406f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5158b16640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f515415b5e0 0x7f515415ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.866 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5158b16640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5154155b70 con 0x7f515406a4d0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5158b16640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515406ee30 con 0x7f515415b5e0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5158b16640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515406f3b0 con 0x7f5154155790 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f515415b5e0 0x7f515415ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5152d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f515406a4d0 0x7f515406a8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42016/0 (socket says 172.21.15.124:42016) 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f515415b5e0 0x7f515415ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58022/0 (socket says 172.21.15.124:58022) 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5152d9d640 1 -- 172.21.15.124:0/2583370709 learned_addr learned my addr 172.21.15.124:0/2583370709 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f515259c640 1 --2- 172.21.15.124:0/2583370709 >> v2:172.21.15.203:3300/0 conn(0x7f5154155790 0x7f515406f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 -- 172.21.15.124:0/2583370709 >> v2:172.21.15.203:3300/0 conn(0x7f5154155790 msgr2=0x7f515406f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 --2- 172.21.15.124:0/2583370709 >> v2:172.21.15.203:3300/0 conn(0x7f5154155790 0x7f515406f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 -- 172.21.15.124:0/2583370709 >> v1:172.21.15.124:6789/0 conn(0x7f515406a4d0 legacy=0x7f515406a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.865+0000 7f5151d9b640 1 -- 172.21.15.124:0/2583370709 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515415c170 con 0x7f515415b5e0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5151d9b640 1 --2- 172.21.15.124:0/2583370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f515415b5e0 0x7f515415ba90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5154072720 tx=0x7f513c02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2c4bb6c38d9d09 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f515159a640 1 -- 172.21.15.124:0/2583370709 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c030710 con 0x7f515415b5e0 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f515159a640 1 -- 172.21.15.124:0/2583370709 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c02e050 con 0x7f515415b5e0 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f515159a640 1 -- 172.21.15.124:0/2583370709 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c030a90 con 0x7f515415b5e0 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5158b16640 1 -- 172.21.15.124:0/2583370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f515415b5e0 msgr2=0x7f515415ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5158b16640 1 --2- 172.21.15.124:0/2583370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f515415b5e0 0x7f515415ba90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f5154072720 tx=0x7f513c02b710 comp rx=0 tx=0).stop 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5158b16640 1 -- 172.21.15.124:0/2583370709 shutdown_connections 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5158b16640 1 --2- 172.21.15.124:0/2583370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f515415b5e0 0x7f515415ba90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.866+0000 7f5158b16640 1 --2- 172.21.15.124:0/2583370709 >> v2:172.21.15.203:3300/0 conn(0x7f5154155790 0x7f515406f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.867+0000 7f5158b16640 1 -- 172.21.15.124:0/2583370709 >> 172.21.15.124:0/2583370709 conn(0x7f51540b1e50 msgr2=0x7f51540b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.867+0000 7f5158b16640 1 -- 172.21.15.124:0/2583370709 shutdown_connections 2023-12-13T23:57:05.868 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.867+0000 7f5158b16640 1 -- 172.21.15.124:0/2583370709 wait complete. 2023-12-13T23:57:05.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.868+0000 7f5158b16640 1 Processor -- start 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.868+0000 7f5158b16640 1 -- start start 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f515406a4d0 0x7f515415adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f515415b5e0 0x7f5154149060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515414a1f0 con 0x7f5154155790 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154159b20 con 0x7f515406a4d0 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5154159d50 con 0x7f515415b5e0 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58026/0 (socket says 172.21.15.124:58026) 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 -- 172.21.15.124:0/3556964058 learned_addr learned my addr 172.21.15.124:0/3556964058 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f515259c640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f515406a4d0 0x7f515415adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 -- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f515415b5e0 msgr2=0x7f5154149060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f515415b5e0 0x7f5154149060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 -- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f515406a4d0 msgr2=0x7f515415adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f515406a4d0 0x7f515415adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 -- 172.21.15.124:0/3556964058 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c009400 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5151d9b640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f513c000c00 tx=0x7f513c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c032890 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c032a10 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515415a030 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c032dc0 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.869+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515415a5f0 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.870+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5154069400 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.871+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f513c039040 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.871+0000 7f514affd640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 0x7f513406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.871+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f513c0ab4e0 con 0x7f5154155790 2023-12-13T23:57:05.872 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.871+0000 7f515259c640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 0x7f513406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:05.873 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.872+0000 7f515259c640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 0x7f513406d680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5144009f30 tx=0x7f514400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:05.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:05.875+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f513c076030 con 0x7f5154155790 2023-12-13T23:57:06.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.082+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f51540b2c50 con 0x7f5154155790 2023-12-13T23:57:06.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.748+0000 7f514affd640 1 -- 172.21.15.124:0/3556964058 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f513c079650 con 0x7f5154155790 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.750+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 msgr2=0x7f513406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.750+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 0x7f513406d680 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f5144009f30 tx=0x7f514400b040 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.750+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 msgr2=0x7f51541595e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.750+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f513c000c00 tx=0x7f513c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 shutdown_connections 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f513406b1c0 0x7f513406d680 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f515415b5e0 0x7f5154149060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5154155790 0x7f51541595e0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 --2- 172.21.15.124:0/3556964058 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f515406a4d0 0x7f515415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 >> 172.21.15.124:0/3556964058 conn(0x7f51540b1e50 msgr2=0x7f51540b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:06.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 shutdown_connections 2023-12-13T23:57:06.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.751+0000 7f5158b16640 1 -- 172.21.15.124:0/3556964058 wait complete. 2023-12-13T23:57:06.763 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-13T23:57:06.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.912+0000 7f8fd62e0640 1 Processor -- start 2023-12-13T23:57:06.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.912+0000 7f8fd62e0640 1 -- start start 2023-12-13T23:57:06.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd62e0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8fd00d2910 0x7f8fd00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd62e0640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00b2b10 con 0x7f8fd00d3730 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd62e0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd00d1590 con 0x7f8fd015d330 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd62e0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd00d3bc0 con 0x7f8fd00d2910 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fcf7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8fd00d2910 0x7f8fd00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd487d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8fd00d3730 0x7f8fd00b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42024/0 (socket says 172.21.15.124:42024) 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fd487d640 1 -- 172.21.15.124:0/2968215448 learned_addr learned my addr 172.21.15.124:0/2968215448 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/2968215448 >> v1:172.21.15.203:6789/0 conn(0x7f8fd015d330 legacy=0x7f8fd00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/2968215448 >> v1:172.21.15.124:6789/0 conn(0x7f8fd00d3730 legacy=0x7f8fd00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/2968215448 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd00d1870 con 0x7f8fd00d2910 2023-12-13T23:57:06.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.913+0000 7f8fceffd640 1 -- 172.21.15.124:0/2968215448 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208608629 0 0) 0x7f8fd00b2b10 con 0x7f8fd00d3730 2023-12-13T23:57:06.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.914+0000 7f8fcf7fe640 1 --2- 172.21.15.124:0/2968215448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d2910 0x7f8fd00d31f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097a0 tx=0x7f8fc402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf49cff410c30b95 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:06.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.914+0000 7f8fceffd640 1 -- 172.21.15.124:0/2968215448 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031830 con 0x7f8fd00d2910 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.914+0000 7f8fceffd640 1 -- 172.21.15.124:0/2968215448 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc4031990 con 0x7f8fd00d2910 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.914+0000 7f8fceffd640 1 -- 172.21.15.124:0/2968215448 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4031d20 con 0x7f8fd00d2910 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 -- 172.21.15.124:0/2968215448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d2910 msgr2=0x7f8fd00d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/2968215448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d2910 0x7f8fd00d31f0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40097a0 tx=0x7f8fc402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 -- 172.21.15.124:0/2968215448 shutdown_connections 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/2968215448 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d2910 0x7f8fd00d31f0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 -- 172.21.15.124:0/2968215448 >> 172.21.15.124:0/2968215448 conn(0x7f8fd0072020 msgr2=0x7f8fd0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 -- 172.21.15.124:0/2968215448 shutdown_connections 2023-12-13T23:57:06.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.915+0000 7f8fd62e0640 1 -- 172.21.15.124:0/2968215448 wait complete. 2023-12-13T23:57:06.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.916+0000 7f8fd62e0640 1 Processor -- start 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.916+0000 7f8fd62e0640 1 -- start start 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 0x7f8fd015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd015d330 0x7f8fd015f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd015bdb0 con 0x7f8fd00d3730 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00cb4e0 con 0x7f8fd015d330 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd62e0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd00cb7c0 con 0x7f8fd00d2910 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58044/0 (socket says 172.21.15.124:58044) 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/1424853106 learned_addr learned my addr 172.21.15.124:0/1424853106 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcffff640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 0x7f8fd015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 msgr2=0x7f8fd015ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fd487d640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd015d330 0x7f8fd015f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 0x7f8fd015ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd015d330 msgr2=0x7f8fd015f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd015d330 0x7f8fd015f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 -- 172.21.15.124:0/1424853106 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc4009400 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcffff640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 0x7f8fd015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.917+0000 7f8fcf7fe640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40098c0 tx=0x7f8fc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.918+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc403ba70 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.918+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc403bbf0 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.918+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd00cbaa0 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.918+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40453f0 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.918+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd00cc060 con 0x7f8fd00d3730 2023-12-13T23:57:06.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.919+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd00cc250 con 0x7f8fd00d3730 2023-12-13T23:57:06.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.919+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8fc404e450 con 0x7f8fd00d3730 2023-12-13T23:57:06.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.919+0000 7f8fccff9640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 0x7f8fbc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:06.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.920+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6190+0+0 (secure 0 0 0) 0x7f8fc40abea0 con 0x7f8fd00d3730 2023-12-13T23:57:06.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.920+0000 7f8fcffff640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 0x7f8fbc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:06.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.920+0000 7f8fcffff640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 0x7f8fbc06d680 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0000f60 tx=0x7f8fc0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:06.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:06.924+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8fc40769f0 con 0x7f8fd00d3730 2023-12-13T23:57:07.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.097+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8fd0069400 con 0x7f8fd00d3730 2023-12-13T23:57:07.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.919+0000 7f8fccff9640 1 -- 172.21.15.124:0/1424853106 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v40) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8fc4035020 con 0x7f8fd00d3730 2023-12-13T23:57:07.918 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' removed 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.922+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 msgr2=0x7f8fbc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.922+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 0x7f8fbc06d680 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f8fc0000f60 tx=0x7f8fc0009040 comp rx=0 tx=0).stop 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 msgr2=0x7f8fd015ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f8fc40098c0 tx=0x7f8fc40046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 shutdown_connections 2023-12-13T23:57:07.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fbc06b1c0 0x7f8fbc06d680 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fd015d330 0x7f8fd015f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fd00d3730 0x7f8fd015ed00 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 --2- 172.21.15.124:0/1424853106 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8fd00d2910 0x7f8fd015ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 >> 172.21.15.124:0/1424853106 conn(0x7f8fd0072020 msgr2=0x7f8fd00b7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 shutdown_connections 2023-12-13T23:57:07.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:07.923+0000 7f8fd62e0640 1 -- 172.21.15.124:0/1424853106 wait complete. 2023-12-13T23:57:07.936 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.046+0000 7f2147e9b640 1 Processor -- start 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.046+0000 7f2147e9b640 1 -- start start 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2147e9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f214015abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2147e9b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214015e2a0 con 0x7f21400b1400 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2147e9b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214015b240 con 0x7f2140143490 2023-12-13T23:57:08.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2147e9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214015b520 con 0x7f214015e730 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2145436640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f21400b1400 0x7f214015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42030/0 (socket says 172.21.15.124:42030) 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2145436640 1 -- 172.21.15.124:0/3978135350 learned_addr learned my addr 172.21.15.124:0/3978135350 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2146438640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f214015abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667404885 0 0) 0x7f214015e2a0 con 0x7f21400b1400 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003520 con 0x7f21400b1400 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627819894 0 0) 0x7f214015b240 con 0x7f2140143490 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.047+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f214015e2a0 con 0x7f2140143490 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1425351303 0 0) 0x7f212c003520 con 0x7f21400b1400 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 msgr2=0x7f214015abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f214015abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 >> v1:172.21.15.203:6789/0 conn(0x7f2140143490 legacy=0x7f21400b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214015dc50 con 0x7f21400b1400 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f2130001670 con 0x7f21400b1400 2023-12-13T23:57:08.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 >> v1:172.21.15.124:6789/0 conn(0x7f21400b1400 legacy=0x7f214015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f212c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f212c0040e0 0x7f212c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f212c004d00 0x7f212c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214015b520 con 0x7f212c004d00 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00bac0 con 0x7f212c0040e0 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f212c00bc80 con 0x7f214015e730 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f212c004d00 0x7f212c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2145c37640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f212c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2145436640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f212c0040e0 0x7f212c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 -- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 msgr2=0x7f212c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f212c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 -- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f212c0040e0 msgr2=0x7f212c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f212c0040e0 0x7f212c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2146438640 1 -- 172.21.15.124:0/3978135350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212c003520 con 0x7f212c004d00 2023-12-13T23:57:08.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2145436640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f212c0040e0 0x7f212c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.048+0000 7f2145c37640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214015e730 0x7f212c003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2146438640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f212c004d00 0x7f212c005180 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2138009fc0 tx=0x7f212c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f16da2ece7e98e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213802cc20 con 0x7f212c004d00 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213800f040 con 0x7f212c004d00 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2144c35640 1 -- 172.21.15.124:0/3978135350 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138034440 con 0x7f212c004d00 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2147e9b640 1 -- 172.21.15.124:0/3978135350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f212c004d00 msgr2=0x7f212c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2147e9b640 1 --2- 172.21.15.124:0/3978135350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f212c004d00 0x7f212c005180 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f2138009fc0 tx=0x7f212c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2146438640 1 -- 172.21.15.124:0/3978135350 reap_dead start 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2147e9b640 1 -- 172.21.15.124:0/3978135350 shutdown_connections 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.049+0000 7f2147e9b640 1 -- 172.21.15.124:0/3978135350 >> 172.21.15.124:0/3978135350 conn(0x7f214006a4d0 msgr2=0x7f214006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.050+0000 7f2147e9b640 1 -- 172.21.15.124:0/3978135350 shutdown_connections 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.050+0000 7f2147e9b640 1 -- 172.21.15.124:0/3978135350 wait complete. 2023-12-13T23:57:08.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 Processor -- start 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 -- start start 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214014c560 0x7f214013fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2140140080 0x7f214013dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214013b350 con 0x7f214013e500 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214013eae0 con 0x7f2140140080 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.051+0000 7f2147e9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214014c940 con 0x7f214014c560 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58060/0 (socket says 172.21.15.124:58060) 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 -- 172.21.15.124:0/1680440573 learned_addr learned my addr 172.21.15.124:0/1680440573 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2145436640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2140140080 0x7f214013dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2145c37640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214014c560 0x7f214013fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 -- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214014c560 msgr2=0x7f214013fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214014c560 0x7f214013fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 -- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2140140080 msgr2=0x7f214013dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2140140080 0x7f214013dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 -- 172.21.15.124:0/1680440573 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214013ed80 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2146438640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2138037040 tx=0x7f2138009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138043070 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2138034ad0 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214013f060 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2138034e40 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.052+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214013f620 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.053+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2140069400 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.054+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f213803e440 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.054+0000 7f21367fc640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 0x7f212406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.054+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5752+0+0 (secure 0 0 0) 0x7f21380ab320 con 0x7f214013e500 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.054+0000 7f2145c37640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 0x7f212406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:08.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.055+0000 7f2145c37640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 0x7f212406d680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f213c001870 tx=0x7f213c0012d0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:08.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.058+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f21380760d0 con 0x7f214013e500 2023-12-13T23:57:08.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.226+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f21400cd400 con 0x7f214013e500 2023-12-13T23:57:08.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.923+0000 7f21367fc640 1 -- 172.21.15.124:0/1680440573 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v41) v1 ==== 169+0+0 (secure 0 0 0) 0x7f21380796f0 con 0x7f214013e500 2023-12-13T23:57:08.922 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' removed 2023-12-13T23:57:08.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 msgr2=0x7f212406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 0x7f212406d680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f213c001870 tx=0x7f213c0012d0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 msgr2=0x7f214014cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f2138037040 tx=0x7f2138009750 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 shutdown_connections 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f212406b1c0 0x7f212406d680 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2140140080 0x7f214013dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f214013e500 0x7f214014cc80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 --2- 172.21.15.124:0/1680440573 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f214014c560 0x7f214013fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 >> 172.21.15.124:0/1680440573 conn(0x7f214006a4d0 msgr2=0x7f214006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 shutdown_connections 2023-12-13T23:57:08.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:08.925+0000 7f2147e9b640 1 -- 172.21.15.124:0/1680440573 wait complete. 2023-12-13T23:57:08.936 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2023-12-13T23:57:09.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.041+0000 7f43e1378640 1 Processor -- start 2023-12-13T23:57:09.040 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.041+0000 7f43e1378640 1 -- start start 2023-12-13T23:57:09.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43e1378640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 0x7f43dc0cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43e1378640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43e1378640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43dc1614e0 con 0x7f43dc15d2d0 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43e1378640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43dc1617c0 con 0x7f43dc160f70 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43e1378640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc161aa0 con 0x7f43dc0ce480 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43d3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 0x7f43dc0cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:36532/0 (socket says 172.21.15.124:36532) 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43daffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f43dc15d2d0 0x7f43dc0b8b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:46030/0 (socket says 172.21.15.124:46030) 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.042+0000 7f43db7fe640 1 -- 172.21.15.124:0/1472544855 learned_addr learned my addr 172.21.15.124:0/1472544855 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43db7fe640 1 -- 172.21.15.124:0/1472544855 >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 msgr2=0x7f43dc0cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43db7fe640 1 --2- 172.21.15.124:0/1472544855 >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 0x7f43dc0cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43db7fe640 1 -- 172.21.15.124:0/1472544855 >> v1:172.21.15.203:6789/0 conn(0x7f43dc15d2d0 legacy=0x7f43dc0b8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43db7fe640 1 -- 172.21.15.124:0/1472544855 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc161d80 con 0x7f43dc0ce480 2023-12-13T23:57:09.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43d3fff640 1 --2- 172.21.15.124:0/1472544855 >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 0x7f43dc0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43db7fe640 1 --2- 172.21.15.124:0/1472544855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009e50 tx=0x7f43d402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9787b3423df73fb3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.043+0000 7f43da7fc640 1 -- 172.21.15.124:0/1472544855 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d402cb00 con 0x7f43dc0ce480 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43da7fc640 1 -- 172.21.15.124:0/1472544855 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d402cc60 con 0x7f43dc0ce480 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43da7fc640 1 -- 172.21.15.124:0/1472544855 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d40343f0 con 0x7f43dc0ce480 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 -- 172.21.15.124:0/1472544855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 msgr2=0x7f43dc0d1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 --2- 172.21.15.124:0/1472544855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009e50 tx=0x7f43d402bf60 comp rx=0 tx=0).stop 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 -- 172.21.15.124:0/1472544855 shutdown_connections 2023-12-13T23:57:09.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 --2- 172.21.15.124:0/1472544855 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc0ce480 0x7f43dc0d1af0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 --2- 172.21.15.124:0/1472544855 >> v2:172.21.15.203:3300/0 conn(0x7f43dc160f70 0x7f43dc0cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 -- 172.21.15.124:0/1472544855 >> 172.21.15.124:0/1472544855 conn(0x7f43dc072230 msgr2=0x7f43dc0cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 -- 172.21.15.124:0/1472544855 shutdown_connections 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.044+0000 7f43e1378640 1 -- 172.21.15.124:0/1472544855 wait complete. 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.045+0000 7f43e1378640 1 Processor -- start 2023-12-13T23:57:09.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.045+0000 7f43e1378640 1 -- start start 2023-12-13T23:57:09.046 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43dc0ce480 0x7f43dc15ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc15d2d0 0x7f43dc14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc1f7400 con 0x7f43dc160f70 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc1f6780 con 0x7f43dc0ce480 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43e1378640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43dc1f6a60 con 0x7f43dc15d2d0 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58072/0 (socket says 172.21.15.124:58072) 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 -- 172.21.15.124:0/2212850862 learned_addr learned my addr 172.21.15.124:0/2212850862 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43daffd640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43dc0ce480 0x7f43dc15ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43d3fff640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc15d2d0 0x7f43dc14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 -- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc15d2d0 msgr2=0x7f43dc14f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc15d2d0 0x7f43dc14f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 -- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43dc0ce480 msgr2=0x7f43dc15ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43dc0ce480 0x7f43dc15ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.046+0000 7f43db7fe640 1 -- 172.21.15.124:0/2212850862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c4009400 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43db7fe640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009e50 tx=0x7f43d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4009510 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d403e450 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d4009b30 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d403e800 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.047+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43dc1f7020 con 0x7f43dc160f70 2023-12-13T23:57:09.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.048+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43dc069400 con 0x7f43dc160f70 2023-12-13T23:57:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.048+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f43d403e960 con 0x7f43dc160f70 2023-12-13T23:57:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.049+0000 7f43d37fe640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 0x7f43bc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.049+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 5338+0+0 (secure 0 0 0) 0x7f43d40ac520 con 0x7f43dc160f70 2023-12-13T23:57:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.049+0000 7f43daffd640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 0x7f43bc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:09.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.049+0000 7f43daffd640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 0x7f43bc06d680 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43cc000f60 tx=0x7f43cc000ed0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:09.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.053+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f43d40773f0 con 0x7f43dc160f70 2023-12-13T23:57:09.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.224+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f43dc0d0aa0 con 0x7f43dc160f70 2023-12-13T23:57:09.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.929+0000 7f43d37fe640 1 -- 172.21.15.124:0/2212850862 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v42) v1 ==== 160+0+0 (secure 0 0 0) 0x7f43d407aa10 con 0x7f43dc160f70 2023-12-13T23:57:09.928 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'data_new' removed 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 msgr2=0x7f43bc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 0x7f43bc06d680 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f43cc000f60 tx=0x7f43cc000ed0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 msgr2=0x7f43dc14f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f43d4009e50 tx=0x7f43d4004a40 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 shutdown_connections 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f43bc06b1c0 0x7f43bc06d680 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f43dc160f70 0x7f43dc14f7e0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f43dc15d2d0 0x7f43dc14f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 --2- 172.21.15.124:0/2212850862 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f43dc0ce480 0x7f43dc15ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 >> 172.21.15.124:0/2212850862 conn(0x7f43dc072230 msgr2=0x7f43dc0d31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.931+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 shutdown_connections 2023-12-13T23:57:09.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:09.932+0000 7f43e1378640 1 -- 172.21.15.124:0/2212850862 wait complete. 2023-12-13T23:57:09.944 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-13T23:57:10.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.056+0000 7f201047c640 1 Processor -- start 2023-12-13T23:57:10.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.056+0000 7f201047c640 1 -- start start 2023-12-13T23:57:10.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f201047c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2008144980 0x7f200815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f201047c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20080b4370 0x7f20080b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f201047c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2008071770 con 0x7f20080ceb70 2023-12-13T23:57:10.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f201047c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20080ccb60 con 0x7f2008144980 2023-12-13T23:57:10.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f201047c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b4ce0 con 0x7f20080b4370 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2008144980 0x7f200815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2008144980 0x7f200815c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58086/0 (socket says 172.21.15.124:58086) 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 -- 172.21.15.124:0/1169386163 learned_addr learned my addr 172.21.15.124:0/1169386163 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 -- 172.21.15.124:0/1169386163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20080b4370 msgr2=0x7f20080b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 --2- 172.21.15.124:0/1169386163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20080b4370 0x7f20080b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 -- 172.21.15.124:0/1169386163 >> v1:172.21.15.203:6789/0 conn(0x7f20080ceb70 legacy=0x7f20080cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:10.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.057+0000 7f200e218640 1 -- 172.21.15.124:0/1169386163 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20080ccdc0 con 0x7f2008144980 2023-12-13T23:57:10.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f200e218640 1 --2- 172.21.15.124:0/1169386163 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2008144980 0x7f200815c7c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2004009870 tx=0x7f200402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565bb5cf648d1ba9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f200d216640 1 -- 172.21.15.124:0/1169386163 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004031640 con 0x7f2008144980 2023-12-13T23:57:10.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f200d216640 1 -- 172.21.15.124:0/1169386163 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20040317a0 con 0x7f2008144980 2023-12-13T23:57:10.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f200d216640 1 -- 172.21.15.124:0/1169386163 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004031b30 con 0x7f2008144980 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f201047c640 1 -- 172.21.15.124:0/1169386163 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2008144980 msgr2=0x7f200815c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.058+0000 7f201047c640 1 --2- 172.21.15.124:0/1169386163 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2008144980 0x7f200815c7c0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f2004009870 tx=0x7f200402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 -- 172.21.15.124:0/1169386163 shutdown_connections 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 --2- 172.21.15.124:0/1169386163 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20080b4370 0x7f20080b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 --2- 172.21.15.124:0/1169386163 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2008144980 0x7f200815c7c0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 -- 172.21.15.124:0/1169386163 >> 172.21.15.124:0/1169386163 conn(0x7f200806a4d0 msgr2=0x7f200806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 -- 172.21.15.124:0/1169386163 shutdown_connections 2023-12-13T23:57:10.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.059+0000 7f201047c640 1 -- 172.21.15.124:0/1169386163 wait complete. 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 Processor -- start 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 -- start start 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 0x7f20080b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2008144980 0x7f20080b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b26a0 con 0x7f20080b4370 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b34e0 con 0x7f20080ceb70 2023-12-13T23:57:10.062 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.060+0000 7f201047c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20080b3eb0 con 0x7f2008144980 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58090/0 (socket says 172.21.15.124:58090) 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 -- 172.21.15.124:0/2426546595 learned_addr learned my addr 172.21.15.124:0/2426546595 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200da17640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 0x7f20080b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200ea19640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2008144980 0x7f20080b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 -- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2008144980 msgr2=0x7f20080b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2008144980 0x7f20080b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 -- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 msgr2=0x7f20080b2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 0x7f20080b2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 -- 172.21.15.124:0/2426546595 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040094d0 con 0x7f20080b4370 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200da17640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 0x7f20080b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f200e218640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2004009870 tx=0x7f20040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200403ac60 con 0x7f20080b4370 2023-12-13T23:57:10.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200403adc0 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2004044630 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20080b28d0 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.061+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20080b2e90 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.062+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2008069400 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.063+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f200402f050 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.063+0000 7f1ffeffd640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 0x7f1fec06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.063+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f20040ab6f0 con 0x7f20080b4370 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.063+0000 7f200da17640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 0x7f1fec06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.064+0000 7f200da17640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 0x7f1fec06d6b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20080b4340 tx=0x7f1ff8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.066 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.067+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2004076720 con 0x7f20080b4370 2023-12-13T23:57:10.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.235+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f200806a9e0 con 0x7f20080b4370 2023-12-13T23:57:10.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.238+0000 7f1ffeffd640 1 -- 172.21.15.124:0/2426546595 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2004079d40 con 0x7f20080b4370 2023-12-13T23:57:10.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 msgr2=0x7f1fec06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 0x7f1fec06d6b0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f20080b4340 tx=0x7f1ff8009040 comp rx=0 tx=0).stop 2023-12-13T23:57:10.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 msgr2=0x7f20080b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2004009870 tx=0x7f20040046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 shutdown_connections 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1fec06b1f0 0x7f1fec06d6b0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20080ceb70 0x7f20080b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2008144980 0x7f20080b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 --2- 172.21.15.124:0/2426546595 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20080b4370 0x7f20080b13e0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 >> 172.21.15.124:0/2426546595 conn(0x7f200806a4d0 msgr2=0x7f2008143c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 shutdown_connections 2023-12-13T23:57:10.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.240+0000 7f201047c640 1 -- 172.21.15.124:0/2426546595 wait complete. 2023-12-13T23:57:10.250 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2023-12-13T23:57:10.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.349+0000 7f5833417640 1 Processor -- start 2023-12-13T23:57:10.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.349+0000 7f5833417640 1 -- start start 2023-12-13T23:57:10.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f5833417640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f582c0d2910 0x7f582c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f5833417640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 0x7f582c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f5833417640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c0b2b10 con 0x7f582c15d330 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f5833417640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c0d1590 con 0x7f582c0d2910 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f5833417640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0d1870 con 0x7f582c0d3730 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f58309b2640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f582c0d2910 0x7f582c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f58311b3640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f582c15d330 0x7f582c0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:42032/0 (socket says 172.21.15.124:42032) 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.350+0000 7f58311b3640 1 -- 172.21.15.124:0/3286832923 learned_addr learned my addr 172.21.15.124:0/3286832923 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58319b4640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 0x7f582c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58309b2640 1 -- 172.21.15.124:0/3286832923 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 msgr2=0x7f582c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58309b2640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 0x7f582c0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58309b2640 1 -- 172.21.15.124:0/3286832923 >> v1:172.21.15.124:6789/0 conn(0x7f582c15d330 legacy=0x7f582c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58309b2640 1 -- 172.21.15.124:0/3286832923 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0d3b90 con 0x7f582c0d2910 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f5823fff640 1 -- 172.21.15.124:0/3286832923 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793596311 0 0) 0x7f582c0b2b10 con 0x7f582c15d330 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58319b4640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 0x7f582c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.351+0000 7f58309b2640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d2910 0x7f582c0d31f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245efd1a1efa201f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5823fff640 1 -- 172.21.15.124:0/3286832923 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c02e070 con 0x7f582c0d2910 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5823fff640 1 -- 172.21.15.124:0/3286832923 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0315a0 con 0x7f582c0d2910 2023-12-13T23:57:10.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5823fff640 1 -- 172.21.15.124:0/3286832923 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c031930 con 0x7f582c0d2910 2023-12-13T23:57:10.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 -- 172.21.15.124:0/3286832923 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d2910 msgr2=0x7f582c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d2910 0x7f582c0d31f0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c02b840 comp rx=0 tx=0).stop 2023-12-13T23:57:10.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 -- 172.21.15.124:0/3286832923 shutdown_connections 2023-12-13T23:57:10.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d3730 0x7f582c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 --2- 172.21.15.124:0/3286832923 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d2910 0x7f582c0d31f0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 -- 172.21.15.124:0/3286832923 >> 172.21.15.124:0/3286832923 conn(0x7f582c072020 msgr2=0x7f582c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 -- 172.21.15.124:0/3286832923 shutdown_connections 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.352+0000 7f5833417640 1 -- 172.21.15.124:0/3286832923 wait complete. 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.353+0000 7f5833417640 1 Processor -- start 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.353+0000 7f5833417640 1 -- start start 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d2910 0x7f582c150580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c15d330 0x7f582c0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0c83a0 con 0x7f582c0d3730 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c15f2a0 con 0x7f582c15d330 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f5833417640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c15f580 con 0x7f582c0d2910 2023-12-13T23:57:10.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:58114/0 (socket says 172.21.15.124:58114) 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 -- 172.21.15.124:0/1541900539 learned_addr learned my addr 172.21.15.124:0/1541900539 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 -- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d2910 msgr2=0x7f582c150580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d2910 0x7f582c150580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 -- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c15d330 msgr2=0x7f582c0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c15d330 0x7f582c0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.354+0000 7f58309b2640 1 -- 172.21.15.124:0/1541900539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c009400 con 0x7f582c0d3730 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f58309b2640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c004230 con 0x7f582c0d3730 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0043b0 con 0x7f582c0d3730 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c15f860 con 0x7f582c0d3730 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c004520 con 0x7f582c0d3730 2023-12-13T23:57:10.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.355+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c15fe20 con 0x7f582c0d3730 2023-12-13T23:57:10.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.356+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c15cee0 con 0x7f582c0d3730 2023-12-13T23:57:10.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.356+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f581c032880 con 0x7f582c0d3730 2023-12-13T23:57:10.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.357+0000 7f5821ffb640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 0x7f581006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:10.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.357+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f581c0ab900 con 0x7f582c0d3730 2023-12-13T23:57:10.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.357+0000 7f58311b3640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 0x7f581006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:10.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.358+0000 7f58311b3640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 0x7f581006d680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5828001560 tx=0x7f5828007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:10.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.361+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f581c0768a0 con 0x7f582c0d3730 2023-12-13T23:57:10.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.589+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) v1 -- 0x7f582c0b74c0 con 0x7f582c0d3730 2023-12-13T23:57:10.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.936+0000 7f5821ffb640 1 -- 172.21.15.124:0/1541900539 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v126) v1 ==== 147+0+0 (secure 0 0 0) 0x7f581c079ec0 con 0x7f582c0d3730 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 msgr2=0x7f581006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 0x7f581006d680 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f5828001560 tx=0x7f5828007ca0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 msgr2=0x7f582c15e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f581c0097a0 tx=0x7f581c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 shutdown_connections 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f581006b1c0 0x7f581006d680 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f582c15d330 0x7f582c0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f582c0d3730 0x7f582c15e930 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 --2- 172.21.15.124:0/1541900539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f582c0d2910 0x7f582c150580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:10.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 >> 172.21.15.124:0/1541900539 conn(0x7f582c072020 msgr2=0x7f582c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:10.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 shutdown_connections 2023-12-13T23:57:10.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:10.938+0000 7f5833417640 1 -- 172.21.15.124:0/1541900539 wait complete. 2023-12-13T23:57:10.949 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2023-12-13T23:57:10.949 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:57:10.949 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:11.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.050+0000 7f218f65d640 1 Processor -- start 2023-12-13T23:57:11.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 -- start start 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21880d2910 0x7f21880d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f21880d3730 0x7f21880b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880b2b10 con 0x7f218815d330 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21880d3bc0 con 0x7f21880d2910 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218f65d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21880d1590 con 0x7f21880d3730 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218cbf8640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21880d2910 0x7f21880d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218d3f9640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f218815d330 0x7f21880b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47560/0 (socket says 172.21.15.124:47560) 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.051+0000 7f218d3f9640 1 -- 172.21.15.124:0/3178449896 learned_addr learned my addr 172.21.15.124:0/3178449896 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218dbfa640 1 --2- 172.21.15.124:0/3178449896 >> v2:172.21.15.203:3300/0 conn(0x7f21880d3730 0x7f21880b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218cbf8640 1 -- 172.21.15.124:0/3178449896 >> v2:172.21.15.203:3300/0 conn(0x7f21880d3730 msgr2=0x7f21880b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:11.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218cbf8640 1 --2- 172.21.15.124:0/3178449896 >> v2:172.21.15.203:3300/0 conn(0x7f21880d3730 0x7f21880b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218cbf8640 1 -- 172.21.15.124:0/3178449896 >> v1:172.21.15.124:6789/0 conn(0x7f218815d330 legacy=0x7f21880b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218cbf8640 1 -- 172.21.15.124:0/3178449896 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21880d1870 con 0x7f21880d2910 2023-12-13T23:57:11.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f217bfff640 1 -- 172.21.15.124:0/3178449896 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287818087 0 0) 0x7f21880b2b10 con 0x7f218815d330 2023-12-13T23:57:11.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.052+0000 7f218cbf8640 1 --2- 172.21.15.124:0/3178449896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21880d2910 0x7f21880d31f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f218806c500 tx=0x7f218402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=948ebda87c527022 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f217bfff640 1 -- 172.21.15.124:0/3178449896 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218402e070 con 0x7f21880d2910 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f217bfff640 1 -- 172.21.15.124:0/3178449896 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21840315a0 con 0x7f21880d2910 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f217bfff640 1 -- 172.21.15.124:0/3178449896 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2184031930 con 0x7f21880d2910 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 -- 172.21.15.124:0/3178449896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21880d2910 msgr2=0x7f21880d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 --2- 172.21.15.124:0/3178449896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21880d2910 0x7f21880d31f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f218806c500 tx=0x7f218402b840 comp rx=0 tx=0).stop 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 -- 172.21.15.124:0/3178449896 shutdown_connections 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 --2- 172.21.15.124:0/3178449896 >> v2:172.21.15.203:3300/0 conn(0x7f21880d3730 0x7f21880b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 --2- 172.21.15.124:0/3178449896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21880d2910 0x7f21880d31f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 -- 172.21.15.124:0/3178449896 >> 172.21.15.124:0/3178449896 conn(0x7f2188072020 msgr2=0x7f2188072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 -- 172.21.15.124:0/3178449896 shutdown_connections 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.053+0000 7f218f65d640 1 -- 172.21.15.124:0/3178449896 wait complete. 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.054+0000 7f218f65d640 1 Processor -- start 2023-12-13T23:57:11.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.054+0000 7f218f65d640 1 -- start start 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21880d2910 0x7f21880c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 0x7f21881f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880c7530 con 0x7f218815d330 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21880c6c40 con 0x7f21880d3730 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218f65d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21880c6ee0 con 0x7f21880d2910 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218dbfa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218dbfa640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45518/0 (socket says 172.21.15.124:45518) 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218dbfa640 1 -- 172.21.15.124:0/1461547286 learned_addr learned my addr 172.21.15.124:0/1461547286 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.055+0000 7f218cbf8640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 0x7f21881f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 -- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21880d2910 msgr2=0x7f21880c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21880d2910 0x7f21880c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 -- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 msgr2=0x7f21881f2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 0x7f21881f2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 -- 172.21.15.124:0/1461547286 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184009400 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218cbf8640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 0x7f21881f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218dbfa640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2180007e40 tx=0x7f218000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2180014620 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218000ee90 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21880c71c0 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21800149e0 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.056+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21881f2740 con 0x7f218815d330 2023-12-13T23:57:11.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.057+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2188069400 con 0x7f218815d330 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.057+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2180014b40 con 0x7f218815d330 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.058+0000 7f2179ffb640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 0x7f216c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.058+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f218008a0b0 con 0x7f218815d330 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.058+0000 7f218d3f9640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 0x7f216c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.058+0000 7f218d3f9640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 0x7f216c06d680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c007960 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.062+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2180055170 con 0x7f218815d330 2023-12-13T23:57:11.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.228+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21880c7670 con 0x7f218815d330 2023-12-13T23:57:11.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.229+0000 7f2179ffb640 1 -- 172.21.15.124:0/1461547286 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+13049 (secure 0 0 0) 0x7f2180058790 con 0x7f218815d330 2023-12-13T23:57:11.228 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:11.228 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":42,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:09.924640+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.124:6835/4122494545":"2023-12-14T23:57:05.746649+0000","172.21.15.124:6834/4122494545":"2023-12-14T23:57:05.746649+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:11.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 msgr2=0x7f216c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 0x7f216c06d680 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f217c007fd0 tx=0x7f217c007960 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 msgr2=0x7f21880c8980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f2180007e40 tx=0x7f218000e4b0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 shutdown_connections 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f216c06b1c0 0x7f216c06d680 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21880d3730 0x7f21881f2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f218815d330 0x7f21880c8980 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 --2- 172.21.15.124:0/1461547286 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21880d2910 0x7f21880c79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 >> 172.21.15.124:0/1461547286 conn(0x7f2188072020 msgr2=0x7f2188072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 shutdown_connections 2023-12-13T23:57:11.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.231+0000 7f218f65d640 1 -- 172.21.15.124:0/1461547286 wait complete. 2023-12-13T23:57:11.241 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:11.242 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:11.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.345+0000 7f0d8f5a9640 1 Processor -- start 2023-12-13T23:57:11.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.346+0000 7f0d8f5a9640 1 -- start start 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.346+0000 7f0d8f5a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0d8815e730 0x7f0d8815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.346+0000 7f0d8f5a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8815e2a0 con 0x7f0d88143490 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.346+0000 7f0d8f5a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8815b240 con 0x7f0d880b1400 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.346+0000 7f0d8f5a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d8815b520 con 0x7f0d8815e730 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8d345640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0d88143490 0x7f0d880b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47566/0 (socket says 172.21.15.124:47566) 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8d345640 1 -- 172.21.15.124:0/576523512 learned_addr learned my addr 172.21.15.124:0/576523512 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647446015 0 0) 0x7f0d8815e2a0 con 0x7f0d88143490 2023-12-13T23:57:11.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> v2:172.21.15.203:3300/0 conn(0x7f0d8815e730 0x7f0d8815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d74003520 con 0x7f0d88143490 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255381124 0 0) 0x7f0d74003520 con 0x7f0d88143490 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 >> v2:172.21.15.203:3300/0 conn(0x7f0d8815e730 msgr2=0x7f0d8815abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 --2- 172.21.15.124:0/576523512 >> v2:172.21.15.203:3300/0 conn(0x7f0d8815e730 0x7f0d8815abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 >> v1:172.21.15.203:6789/0 conn(0x7f0d880b1400 legacy=0x7f0d8815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d8815dc50 con 0x7f0d88143490 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0d80001670 con 0x7f0d88143490 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> v2:172.21.15.203:3300/0 conn(0x7f0d8815e730 0x7f0d8815abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 >> v1:172.21.15.124:6789/0 conn(0x7f0d88143490 legacy=0x7f0d880b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d74003970 0x7f0d74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d740047c0 0x7f0d74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8815b520 con 0x7f0d74005180 2023-12-13T23:57:11.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7400c0d0 con 0x7f0d74003970 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d7400c290 con 0x7f0d740047c0 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8d345640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d74003970 0x7f0d74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8cb44640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d740047c0 0x7f0d74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 -- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d740047c0 msgr2=0x7f0d74004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d740047c0 0x7f0d74004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 -- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d74003970 msgr2=0x7f0d74003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d74003970 0x7f0d74003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.347+0000 7f0d8db46640 1 -- 172.21.15.124:0/576523512 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d74003520 con 0x7f0d74005180 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.348+0000 7f0d8db46640 1 -- 172.21.15.124:0/576523512 reap_dead start 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.348+0000 7f0d8db46640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d7400b980 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0d840073e0 tx=0x7f0d740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f11c1b1f9404bf0f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.348+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8402cdc0 con 0x7f0d74005180 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.348+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d84034440 con 0x7f0d74005180 2023-12-13T23:57:11.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.348+0000 7f0d7ffff640 1 -- 172.21.15.124:0/576523512 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d840347d0 con 0x7f0d74005180 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 msgr2=0x7f0d7400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d7400b980 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f0d840073e0 tx=0x7f0d740057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/576523512 shutdown_connections 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/576523512 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d7400b980 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/576523512 >> 172.21.15.124:0/576523512 conn(0x7f0d8806a4d0 msgr2=0x7f0d8806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/576523512 shutdown_connections 2023-12-13T23:57:11.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.349+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/576523512 wait complete. 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.350+0000 7f0d8f5a9640 1 Processor -- start 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.350+0000 7f0d8f5a9640 1 -- start start 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d880717f0 0x7f0d88071c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d88151dd0 0x7f0d88152250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d881517c0 con 0x7f0d74005180 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d880721b0 con 0x7f0d880717f0 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d88136050 con 0x7f0d88151dd0 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45534/0 (socket says 172.21.15.124:45534) 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 -- 172.21.15.124:0/3133817723 learned_addr learned my addr 172.21.15.124:0/3133817723 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 -- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d88151dd0 msgr2=0x7f0d88152250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d88151dd0 0x7f0d88152250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 -- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d880717f0 msgr2=0x7f0d88071c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d880717f0 0x7f0d88071c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 -- 172.21.15.124:0/3133817723 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d88135720 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8d345640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0d800072c0 tx=0x7f0d80007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80001660 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8000f960 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d88135a00 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.351+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8000fd10 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.352+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d88152b60 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.353+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d88069400 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.353+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0d8001d460 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.353+0000 7f0d7dffb640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 0x7f0d6c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.353+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0d800897d0 con 0x7f0d74005180 2023-12-13T23:57:11.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.354+0000 7f0d8cb44640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 0x7f0d6c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.354+0000 7f0d8cb44640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 0x7f0d6c06d6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0d881431f0 tx=0x7f0d78009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.358+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0d80054890 con 0x7f0d74005180 2023-12-13T23:57:11.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.523+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0d880b2820 con 0x7f0d74005180 2023-12-13T23:57:11.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.524+0000 7f0d7dffb640 1 -- 172.21.15.124:0/3133817723 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0d80057eb0 con 0x7f0d74005180 2023-12-13T23:57:11.523 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 msgr2=0x7f0d6c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 0x7f0d6c06d6b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f0d881431f0 tx=0x7f0d78009040 comp rx=0 tx=0).stop 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 msgr2=0x7f0d88151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f0d800072c0 tx=0x7f0d80007780 comp rx=0 tx=0).stop 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 shutdown_connections 2023-12-13T23:57:11.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0d6c06b1f0 0x7f0d6c06d6b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0d880717f0 0x7f0d88071c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0d88151dd0 0x7f0d88152250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 --2- 172.21.15.124:0/3133817723 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d74005180 0x7f0d88151280 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 >> 172.21.15.124:0/3133817723 conn(0x7f0d8806a4d0 msgr2=0x7f0d880afa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 shutdown_connections 2023-12-13T23:57:11.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.526+0000 7f0d8f5a9640 1 -- 172.21.15.124:0/3133817723 wait complete. 2023-12-13T23:57:11.536 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2023-12-13T23:57:11.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.635+0000 7f8253e90640 1 Processor -- start 2023-12-13T23:57:11.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 -- start start 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 0x7f824c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c0b49f0 con 0x7f824c150560 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f824c0b3560 con 0x7f824c150940 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.636+0000 7f8253e90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824c0b3de0 con 0x7f824c15dec0 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825142b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 0x7f824c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.636 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f8251c2c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f824c150560 0x7f824c0b2e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:34080/0 (socket says 172.21.15.124:34080) 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f8251c2c640 1 -- 172.21.15.124:0/3536098231 learned_addr learned my addr 172.21.15.124:0/3536098231 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:43460/0 (socket says 172.21.15.124:43460) 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 -- 172.21.15.124:0/3536098231 >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 msgr2=0x7f824c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 --2- 172.21.15.124:0/3536098231 >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 0x7f824c15d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 -- 172.21.15.124:0/3536098231 >> v1:172.21.15.203:6789/0 conn(0x7f824c150560 legacy=0x7f824c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825242d640 1 -- 172.21.15.124:0/3536098231 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c0b40c0 con 0x7f824c15dec0 2023-12-13T23:57:11.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.637+0000 7f825142b640 1 --2- 172.21.15.124:0/3536098231 >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 0x7f824c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:11.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.638+0000 7f825242d640 1 --2- 172.21.15.124:0/3536098231 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8244009d90 tx=0x7f824402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=440aca72b0b868fd server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.638+0000 7f8250c2a640 1 -- 172.21.15.124:0/3536098231 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f824402cc50 con 0x7f824c15dec0 2023-12-13T23:57:11.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.638+0000 7f8250c2a640 1 -- 172.21.15.124:0/3536098231 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f824402cdb0 con 0x7f824c15dec0 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.638+0000 7f8250c2a640 1 -- 172.21.15.124:0/3536098231 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244034660 con 0x7f824c15dec0 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 -- 172.21.15.124:0/3536098231 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 msgr2=0x7f824c15d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 --2- 172.21.15.124:0/3536098231 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8244009d90 tx=0x7f824402bea0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 -- 172.21.15.124:0/3536098231 shutdown_connections 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 --2- 172.21.15.124:0/3536098231 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c15dec0 0x7f824c15d740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 --2- 172.21.15.124:0/3536098231 >> v2:172.21.15.203:3300/0 conn(0x7f824c150940 0x7f824c15d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 -- 172.21.15.124:0/3536098231 >> 172.21.15.124:0/3536098231 conn(0x7f824c06a4d0 msgr2=0x7f824c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 -- 172.21.15.124:0/3536098231 shutdown_connections 2023-12-13T23:57:11.639 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.639+0000 7f8253e90640 1 -- 172.21.15.124:0/3536098231 wait complete. 2023-12-13T23:57:11.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.640+0000 7f8253e90640 1 Processor -- start 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.640+0000 7f8253e90640 1 -- start start 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824c15dec0 0x7f824c070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 0x7f824c150f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c071780 con 0x7f824c150560 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824c151c90 con 0x7f824c15dec0 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824c151f70 con 0x7f824c071300 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45550/0 (socket says 172.21.15.124:45550) 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 -- 172.21.15.124:0/4053351737 learned_addr learned my addr 172.21.15.124:0/4053351737 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f825142b640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824c15dec0 0x7f824c070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f825242d640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 0x7f824c150f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 -- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 msgr2=0x7f824c150f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 0x7f824c150f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 -- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824c15dec0 msgr2=0x7f824c070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824c15dec0 0x7f824c070dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 -- 172.21.15.124:0/4053351737 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c009400 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f825242d640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 0x7f824c150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8251c2c640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f824800c4a0 tx=0x7f824800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248005ce0 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8248005e40 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8248007270 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8244009a70 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.641+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c152530 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.642+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824c069400 con 0x7f824c150560 2023-12-13T23:57:11.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.643+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8248007490 con 0x7f824c150560 2023-12-13T23:57:11.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.643+0000 7f82427fc640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 0x7f823006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:11.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.643+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f8248088b60 con 0x7f824c150560 2023-12-13T23:57:11.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.644+0000 7f825142b640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 0x7f823006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:11.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.644+0000 7f825142b640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 0x7f823006d680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f824c1560d0 tx=0x7f823c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:11.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.648+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8248053c40 con 0x7f824c150560 2023-12-13T23:57:11.878 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.878+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7f824c0ce930 con 0x7f824c150560 2023-12-13T23:57:11.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.937+0000 7f82427fc640 1 -- 172.21.15.124:0/4053351737 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v127) v1 ==== 137+0+0 (secure 0 0 0) 0x7f8248017020 con 0x7f824c150560 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.938+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 msgr2=0x7f823006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 0x7f823006d680 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f824c1560d0 tx=0x7f823c008040 comp rx=0 tx=0).stop 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 msgr2=0x7f824c14fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f824800c4a0 tx=0x7f824800c960 comp rx=0 tx=0).stop 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 shutdown_connections 2023-12-13T23:57:11.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f823006b1c0 0x7f823006d680 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f824c15dec0 0x7f824c070dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f824c071300 0x7f824c150f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 --2- 172.21.15.124:0/4053351737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f824c150560 0x7f824c14fc30 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 >> 172.21.15.124:0/4053351737 conn(0x7f824c06a4d0 msgr2=0x7f824c15d150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 shutdown_connections 2023-12-13T23:57:11.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:11.939+0000 7f8253e90640 1 -- 172.21.15.124:0/4053351737 wait complete. 2023-12-13T23:57:11.949 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-12-13T23:57:12.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.052+0000 7fe48db61640 1 Processor -- start 2023-12-13T23:57:12.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.052+0000 7fe48db61640 1 -- start start 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe48815d330 0x7fe4880b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4880d2910 0x7fe4880d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 0x7fe4880b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4880b2b10 con 0x7fe4880d2910 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4880d3b90 con 0x7fe48815d330 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe48db61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4880d1590 con 0x7fe4880d3730 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe47effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4880d2910 0x7fe4880d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe47effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4880d2910 0x7fe4880d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45558/0 (socket says 172.21.15.124:45558) 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.053+0000 7fe47effd640 1 -- 172.21.15.124:0/1122904313 learned_addr learned my addr 172.21.15.124:0/1122904313 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe4877fe640 1 --2- 172.21.15.124:0/1122904313 >> v2:172.21.15.203:3300/0 conn(0x7fe48815d330 0x7fe4880b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe487fff640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 0x7fe4880b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 -- 172.21.15.124:0/1122904313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 msgr2=0x7fe4880b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:12.054 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 0x7fe4880b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 -- 172.21.15.124:0/1122904313 >> v2:172.21.15.203:3300/0 conn(0x7fe48815d330 msgr2=0x7fe4880b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 --2- 172.21.15.124:0/1122904313 >> v2:172.21.15.203:3300/0 conn(0x7fe48815d330 0x7fe4880b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 -- 172.21.15.124:0/1122904313 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4880d1870 con 0x7fe4880d2910 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe487fff640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 0x7fe4880b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.054+0000 7fe47effd640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4880d2910 0x7fe4880d31f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe4880d0d10 tx=0x7fe47002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81693134bc9866f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe486ffd640 1 -- 172.21.15.124:0/1122904313 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4700317e0 con 0x7fe4880d2910 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe486ffd640 1 -- 172.21.15.124:0/1122904313 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe470031940 con 0x7fe4880d2910 2023-12-13T23:57:12.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe486ffd640 1 -- 172.21.15.124:0/1122904313 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470031cd0 con 0x7fe4880d2910 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 -- 172.21.15.124:0/1122904313 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4880d2910 msgr2=0x7fe4880d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4880d2910 0x7fe4880d31f0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fe4880d0d10 tx=0x7fe47002b880 comp rx=0 tx=0).stop 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 -- 172.21.15.124:0/1122904313 shutdown_connections 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4880d2910 0x7fe4880d31f0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 --2- 172.21.15.124:0/1122904313 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d3730 0x7fe4880b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 --2- 172.21.15.124:0/1122904313 >> v2:172.21.15.203:3300/0 conn(0x7fe48815d330 0x7fe4880b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 -- 172.21.15.124:0/1122904313 >> 172.21.15.124:0/1122904313 conn(0x7fe488072020 msgr2=0x7fe488072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:12.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 -- 172.21.15.124:0/1122904313 shutdown_connections 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.055+0000 7fe48db61640 1 -- 172.21.15.124:0/1122904313 wait complete. 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.056+0000 7fe48db61640 1 Processor -- start 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.056+0000 7fe48db61640 1 -- start start 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d2910 0x7fe488150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 0x7fe48815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48815fce0 con 0x7fe48815d330 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48815ff20 con 0x7fe4880d3730 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe48db61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe488160200 con 0x7fe4880d2910 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe487fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe487fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45560/0 (socket says 172.21.15.124:45560) 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.057+0000 7fe487fff640 1 -- 172.21.15.124:0/1470866334 learned_addr learned my addr 172.21.15.124:0/1470866334 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 -- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d2910 msgr2=0x7fe488150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d2910 0x7fe488150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe47effd640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 0x7fe48815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 -- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 msgr2=0x7fe48815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 0x7fe48815f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 -- 172.21.15.124:0/1470866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700094d0 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe47effd640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 0x7fe48815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe487fff640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe48000cef0 tx=0x7fe48000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000d840 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48000d9a0 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4881604e0 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48000dd50 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.058+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4880cb380 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.059+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4880b5e10 con 0x7fe48815d330 2023-12-13T23:57:12.060 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.059+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe48001e3b0 con 0x7fe48815d330 2023-12-13T23:57:12.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.060+0000 7fe484ff9640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 0x7fe46806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.060+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe48008a150 con 0x7fe48815d330 2023-12-13T23:57:12.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.060+0000 7fe4877fe640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 0x7fe46806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.061 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.060+0000 7fe4877fe640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 0x7fe46806d680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe4780098f0 tx=0x7fe478008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.064+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe480055180 con 0x7fe48815d330 2023-12-13T23:57:12.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.231+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe4880cb570 con 0x7fe48815d330 2023-12-13T23:57:12.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.236+0000 7fe484ff9640 1 -- 172.21.15.124:0/1470866334 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe4800587a0 con 0x7fe48815d330 2023-12-13T23:57:12.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 msgr2=0x7fe46806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 0x7fe46806d680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe4780098f0 tx=0x7fe478008040 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 msgr2=0x7fe48814c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fe48000cef0 tx=0x7fe48000a710 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 shutdown_connections 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe46806b1c0 0x7fe46806d680 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4880d3730 0x7fe48815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe48815d330 0x7fe48814c940 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 --2- 172.21.15.124:0/1470866334 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4880d2910 0x7fe488150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 >> 172.21.15.124:0/1470866334 conn(0x7fe488072020 msgr2=0x7fe4880b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 shutdown_connections 2023-12-13T23:57:12.238 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.238+0000 7fe48db61640 1 -- 172.21.15.124:0/1470866334 wait complete. 2023-12-13T23:57:12.248 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-13T23:57:12.248 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:12.261 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:57:12.293 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:57:12.300 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:12.378 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-13T23:57:12.379 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:12.379 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:12.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:12.394 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:12.394 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:12.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:12.435 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:12.435 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:12.450 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:12.450 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:12.450 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:12.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:12.492 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-12-13T23:57:12.492 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:12.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.592+0000 7fd385cad640 1 Processor -- start 2023-12-13T23:57:12.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.592+0000 7fd385cad640 1 -- start start 2023-12-13T23:57:12.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd385cad640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd3800d2910 0x7fd3800d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd385cad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d3730 0x7fd3800b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd385cad640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3800b2b10 con 0x7fd38015d330 2023-12-13T23:57:12.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd385cad640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3800d1590 con 0x7fd3800d2910 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd385cad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3800d1870 con 0x7fd3800d3730 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd37effd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd3800d2910 0x7fd3800d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd37f7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd38015d330 0x7fd3800b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47572/0 (socket says 172.21.15.124:47572) 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.593+0000 7fd37f7fe640 1 -- 172.21.15.124:0/3443079588 learned_addr learned my addr 172.21.15.124:0/3443079588 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37e7fc640 1 -- 172.21.15.124:0/3443079588 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397475026 0 0) 0x7fd3800b2b10 con 0x7fd38015d330 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37e7fc640 1 -- 172.21.15.124:0/3443079588 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd368003520 con 0x7fd38015d330 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37effd640 1 -- 172.21.15.124:0/3443079588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d3730 msgr2=0x7fd3800b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37effd640 1 --2- 172.21.15.124:0/3443079588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d3730 0x7fd3800b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37effd640 1 -- 172.21.15.124:0/3443079588 >> v1:172.21.15.124:6789/0 conn(0x7fd38015d330 legacy=0x7fd3800b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37effd640 1 -- 172.21.15.124:0/3443079588 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3800d3b90 con 0x7fd3800d2910 2023-12-13T23:57:12.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37effd640 1 --2- 172.21.15.124:0/3443079588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3800d2910 0x7fd3800d31f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097a0 tx=0x7fd37402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2c09a0ffa86214b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.594+0000 7fd37e7fc640 1 -- 172.21.15.124:0/3443079588 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37402e070 con 0x7fd3800d2910 2023-12-13T23:57:12.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd37e7fc640 1 -- 172.21.15.124:0/3443079588 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3740315a0 con 0x7fd3800d2910 2023-12-13T23:57:12.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd37e7fc640 1 -- 172.21.15.124:0/3443079588 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd374031910 con 0x7fd3800d2910 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 -- 172.21.15.124:0/3443079588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3800d2910 msgr2=0x7fd3800d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 --2- 172.21.15.124:0/3443079588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3800d2910 0x7fd3800d31f0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd3740097a0 tx=0x7fd37402b840 comp rx=0 tx=0).stop 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 -- 172.21.15.124:0/3443079588 shutdown_connections 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 --2- 172.21.15.124:0/3443079588 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d3730 0x7fd3800b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 --2- 172.21.15.124:0/3443079588 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3800d2910 0x7fd3800d31f0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 -- 172.21.15.124:0/3443079588 >> 172.21.15.124:0/3443079588 conn(0x7fd380072020 msgr2=0x7fd380072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 -- 172.21.15.124:0/3443079588 shutdown_connections 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.595+0000 7fd385cad640 1 -- 172.21.15.124:0/3443079588 wait complete. 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.596+0000 7fd385cad640 1 Processor -- start 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.596+0000 7fd385cad640 1 -- start start 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d2910 0x7fd3800c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 0x7fd3800b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38015f3a0 con 0x7fd38015d330 2023-12-13T23:57:12.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd38015e760 con 0x7fd3800d3730 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd385cad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd38015ea00 con 0x7fd3800d2910 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45582/0 (socket says 172.21.15.124:45582) 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37ffff640 1 -- 172.21.15.124:0/3984457805 learned_addr learned my addr 172.21.15.124:0/3984457805 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37f7fe640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d2910 0x7fd3800c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37effd640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 0x7fd3800b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37ffff640 1 -- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d2910 msgr2=0x7fd3800c8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.597+0000 7fd37ffff640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d2910 0x7fd3800c8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd37ffff640 1 -- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 msgr2=0x7fd3800b6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd37ffff640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 0x7fd3800b6a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd37ffff640 1 -- 172.21.15.124:0/3984457805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd374009400 con 0x7fd38015d330 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd37effd640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 0x7fd3800b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd37ffff640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd37800b970 tx=0x7fd37800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd37800cb40 con 0x7fd38015d330 2023-12-13T23:57:12.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd37800ccc0 con 0x7fd38015d330 2023-12-13T23:57:12.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd38015ece0 con 0x7fd38015d330 2023-12-13T23:57:12.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd378014680 con 0x7fd38015d330 2023-12-13T23:57:12.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.598+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd38015f7c0 con 0x7fd38015d330 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.599+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd38015f4e0 con 0x7fd38015d330 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.599+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd37801d3b0 con 0x7fd38015d330 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.600+0000 7fd367fff640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 0x7fd36c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.600+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fd378089530 con 0x7fd38015d330 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.600+0000 7fd37f7fe640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 0x7fd36c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.601 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.600+0000 7fd37f7fe640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 0x7fd36c06d6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd370005880 tx=0x7fd370009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.603 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.604+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd38015f4e0 con 0x7fd38015d330 2023-12-13T23:57:12.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.805+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd38015f4e0 con 0x7fd38015d330 2023-12-13T23:57:12.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.805+0000 7fd367fff640 1 -- 172.21.15.124:0/3984457805 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3707 (secure 0 0 0) 0x7fd38015f4e0 con 0x7fd38015d330 2023-12-13T23:57:12.805 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:12.805 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4769,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/4066436795","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":4066436795},{"type":"v1","addr":"172.21.15.203:6836","nonce":4066436795}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":4772,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/216517111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":216517111},{"type":"v1","addr":"172.21.15.203:6837","nonce":216517111}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":5247,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/3674956093","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":3674956093},{"type":"v1","addr":"172.21.15.124:6833","nonce":3674956093}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5817,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/1448489898","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1448489898},{"type":"v1","addr":"172.21.15.124:6835","nonce":1448489898}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-12-13T23:57:12.805 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 17 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 msgr2=0x7fd36c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 0x7fd36c06d6b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd370005880 tx=0x7fd370009040 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 msgr2=0x7fd3800b6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd37800b970 tx=0x7fd37800be30 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 shutdown_connections 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd36c06b1f0 0x7fd36c06d6b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd3800d3730 0x7fd3800b6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd38015d330 0x7fd3800b6f70 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 --2- 172.21.15.124:0/3984457805 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd3800d2910 0x7fd3800c8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 >> 172.21.15.124:0/3984457805 conn(0x7fd380072020 msgr2=0x7fd380072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 shutdown_connections 2023-12-13T23:57:12.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.807+0000 7fd385cad640 1 -- 172.21.15.124:0/3984457805 wait complete. 2023-12-13T23:57:12.818 INFO:tasks.ceph.mds.a:Restarting daemon 2023-12-13T23:57:12.818 INFO:tasks.ceph.mds.a:Stopping old one... 2023-12-13T23:57:12.818 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-13T23:57:12.818 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:12.818 INFO:tasks.ceph.mds.c:Restarting daemon 2023-12-13T23:57:12.818 INFO:tasks.ceph.mds.c:Stopping old one... 2023-12-13T23:57:12.819 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-13T23:57:12.819 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:12.819 INFO:tasks.ceph.mds.b:Restarting daemon 2023-12-13T23:57:12.819 INFO:tasks.ceph.mds.b:Stopping old one... 2023-12-13T23:57:12.819 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-13T23:57:12.819 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:12.819 INFO:tasks.ceph.mds.d:Restarting daemon 2023-12-13T23:57:12.819 INFO:tasks.ceph.mds.d:Stopping old one... 2023-12-13T23:57:12.821 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-13T23:57:12.821 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:12.828 INFO:tasks.ceph.mds.a:Stopped 2023-12-13T23:57:12.828 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-12-13T23:57:12.829 INFO:tasks.ceph.mds.c:Stopped 2023-12-13T23:57:12.829 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-12-13T23:57:12.831 INFO:tasks.ceph.mds.b:Stopped 2023-12-13T23:57:12.831 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-12-13T23:57:12.832 INFO:tasks.ceph.mds.d:Stopped 2023-12-13T23:57:12.832 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-12-13T23:57:12.871 INFO:tasks.ceph.mds.a:Started 2023-12-13T23:57:12.871 INFO:tasks.ceph.mds.c:Started 2023-12-13T23:57:12.872 DEBUG:teuthology.parallel:result is None 2023-12-13T23:57:12.872 DEBUG:teuthology.parallel:result is None 2023-12-13T23:57:12.875 INFO:tasks.ceph.mds.b:Started 2023-12-13T23:57:12.876 DEBUG:teuthology.parallel:result is None 2023-12-13T23:57:12.876 INFO:tasks.ceph.mds.d:Started 2023-12-13T23:57:12.876 DEBUG:teuthology.parallel:result is None 2023-12-13T23:57:12.877 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-12-13T23:57:12.948 INFO:tasks.ceph.mds.a.smithi124.stdout:starting mds.a at 2023-12-13T23:57:12.950 INFO:tasks.ceph.mds.c.smithi124.stdout:starting mds.c at 2023-12-13T23:57:12.961 INFO:tasks.ceph.mds.d.smithi203.stdout:starting mds.d at 2023-12-13T23:57:12.961 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2023-12-13T23:57:12.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.982+0000 7f0a4c0cf640 1 Processor -- start 2023-12-13T23:57:12.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 -- start start 2023-12-13T23:57:12.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 0x7f0a4406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0a44160fb0 0x7f0a44161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a44161d20 con 0x7f0a440b3ff0 2023-12-13T23:57:12.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a440d1ee0 con 0x7f0a44160fb0 2023-12-13T23:57:12.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.983+0000 7f0a4c0cf640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4406f370 con 0x7f0a4406ea50 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0a44160fb0 0x7f0a44161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0a44160fb0 0x7f0a44161440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45610/0 (socket says 172.21.15.124:45610) 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 -- 172.21.15.124:0/2531041330 learned_addr learned my addr 172.21.15.124:0/2531041330 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a49e6b640 1 --2- 172.21.15.124:0/2531041330 >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 0x7f0a4406ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 -- 172.21.15.124:0/2531041330 >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 msgr2=0x7f0a4406ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 --2- 172.21.15.124:0/2531041330 >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 0x7f0a4406ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 -- 172.21.15.124:0/2531041330 >> v1:172.21.15.203:6789/0 conn(0x7f0a440b3ff0 legacy=0x7f0a440b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a4966a640 1 -- 172.21.15.124:0/2531041330 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4415e760 con 0x7f0a44160fb0 2023-12-13T23:57:12.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.984+0000 7f0a49e6b640 1 --2- 172.21.15.124:0/2531041330 >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 0x7f0a4406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4966a640 1 --2- 172.21.15.124:0/2531041330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a44160fb0 0x7f0a44161440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0a4406f680 tx=0x7f0a3402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2167a5363181092c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a48e69640 1 -- 172.21.15.124:0/2531041330 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34031870 con 0x7f0a44160fb0 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a48e69640 1 -- 172.21.15.124:0/2531041330 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a340319d0 con 0x7f0a44160fb0 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a48e69640 1 -- 172.21.15.124:0/2531041330 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a34031d60 con 0x7f0a44160fb0 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/2531041330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a44160fb0 msgr2=0x7f0a44161440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/2531041330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a44160fb0 0x7f0a44161440 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0a4406f680 tx=0x7f0a3402b910 comp rx=0 tx=0).stop 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/2531041330 shutdown_connections 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/2531041330 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a44160fb0 0x7f0a44161440 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/2531041330 >> v2:172.21.15.203:3300/0 conn(0x7f0a4406ea50 0x7f0a4406ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.985+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/2531041330 >> 172.21.15.124:0/2531041330 conn(0x7f0a4406a6b0 msgr2=0x7f0a4406aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.986+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/2531041330 shutdown_connections 2023-12-13T23:57:12.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.986+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/2531041330 wait complete. 2023-12-13T23:57:12.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 Processor -- start 2023-12-13T23:57:12.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 -- start start 2023-12-13T23:57:12.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a4406ea50 0x7f0a4413e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a44160fb0 0x7f0a44159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4415abf0 con 0x7f0a440b3ff0 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4415a630 con 0x7f0a4406ea50 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.987+0000 7f0a4c0cf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a4415a910 con 0x7f0a44160fb0 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45622/0 (socket says 172.21.15.124:45622) 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 -- 172.21.15.124:0/760978363 learned_addr learned my addr 172.21.15.124:0/760978363 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a49e6b640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a4406ea50 0x7f0a4413e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4a66c640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a44160fb0 0x7f0a44159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 -- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a44160fb0 msgr2=0x7f0a44159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a44160fb0 0x7f0a44159bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 -- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a4406ea50 msgr2=0x7f0a4413e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a4406ea50 0x7f0a4413e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 -- 172.21.15.124:0/760978363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a340094d0 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4966a640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0a3402b950 tx=0x7f0a340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a3403ad30 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a340443f0 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a4413d970 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a340447a0 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.988+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a4413f590 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.989+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a4413d7d0 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.990+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0a34044900 con 0x7f0a440b3ff0 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.990+0000 7f0a3a7fc640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 0x7f0a2806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:12.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.990+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0a3402d040 con 0x7f0a440b3ff0 2023-12-13T23:57:12.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.990+0000 7f0a49e6b640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 0x7f0a2806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:12.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.991+0000 7f0a49e6b640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 0x7f0a2806d680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0a40006e80 tx=0x7f0a40008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:12.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:12.994+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0a4413d7d0 con 0x7f0a440b3ff0 2023-12-13T23:57:13.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.173+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f0a4413de90 con 0x7f0a440b3ff0 2023-12-13T23:57:13.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.958+0000 7f0a3a7fc640 1 -- 172.21.15.124:0/760978363 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v44) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0a4413de90 con 0x7f0a440b3ff0 2023-12-13T23:57:13.958 INFO:teuthology.orchestra.run.smithi124.stderr: removed all blocklist entries 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.960+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 msgr2=0x7f0a2806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.960+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 0x7f0a2806d680 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f0a40006e80 tx=0x7f0a40008040 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.960+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 msgr2=0x7f0a4406c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.960+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f0a3402b950 tx=0x7f0a340046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 shutdown_connections 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a2806b1c0 0x7f0a2806d680 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a44160fb0 0x7f0a44159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a440b3ff0 0x7f0a4406c8a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 --2- 172.21.15.124:0/760978363 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a4406ea50 0x7f0a4413e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:13.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 >> 172.21.15.124:0/760978363 conn(0x7f0a4406a6b0 msgr2=0x7f0a440b4460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:13.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 shutdown_connections 2023-12-13T23:57:13.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:13.961+0000 7f0a4c0cf640 1 -- 172.21.15.124:0/760978363 wait complete. 2023-12-13T23:57:13.972 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-12-13T23:57:14.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.073+0000 7fbf9742b640 1 Processor -- start 2023-12-13T23:57:14.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 -- start start 2023-12-13T23:57:14.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbf90143490 0x7fbf900b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 0x7fbf900b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 0x7fbf9015a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9015b520 con 0x7fbf90143490 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf90143870 con 0x7fbf9015e730 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.074+0000 7fbf9742b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9015e2a0 con 0x7fbf9015a4e0 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbf90143490 0x7fbf900b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbf90143490 0x7fbf900b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45626/0 (socket says 172.21.15.124:45626) 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 -- 172.21.15.124:0/1898840574 learned_addr learned my addr 172.21.15.124:0/1898840574 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf959c8640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 0x7fbf9015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf949c6640 1 --2- 172.21.15.124:0/1898840574 >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 0x7fbf900b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 -- 172.21.15.124:0/1898840574 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 msgr2=0x7fbf9015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 0x7fbf9015a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 -- 172.21.15.124:0/1898840574 >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 msgr2=0x7fbf900b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 --2- 172.21.15.124:0/1898840574 >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 0x7fbf900b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf951c7640 1 -- 172.21.15.124:0/1898840574 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9015b240 con 0x7fbf90143490 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf959c8640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 0x7fbf9015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.075+0000 7fbf949c6640 1 --2- 172.21.15.124:0/1898840574 >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 0x7fbf900b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf951c7640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf90143490 0x7fbf900b0cf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbf900aeca0 tx=0x7fbf8c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2470bbf338c455eb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf87fff640 1 -- 172.21.15.124:0/1898840574 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c02ee90 con 0x7fbf90143490 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf87fff640 1 -- 172.21.15.124:0/1898840574 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c036440 con 0x7fbf90143490 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf87fff640 1 -- 172.21.15.124:0/1898840574 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0367d0 con 0x7fbf90143490 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 -- 172.21.15.124:0/1898840574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf90143490 msgr2=0x7fbf900b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf90143490 0x7fbf900b0cf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fbf900aeca0 tx=0x7fbf8c02e0e0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 -- 172.21.15.124:0/1898840574 shutdown_connections 2023-12-13T23:57:14.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf9015a4e0 0x7fbf9015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 --2- 172.21.15.124:0/1898840574 >> v2:172.21.15.203:3300/0 conn(0x7fbf9015e730 0x7fbf900b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 --2- 172.21.15.124:0/1898840574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf90143490 0x7fbf900b0cf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 -- 172.21.15.124:0/1898840574 >> 172.21.15.124:0/1898840574 conn(0x7fbf9006a4d0 msgr2=0x7fbf9006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 -- 172.21.15.124:0/1898840574 shutdown_connections 2023-12-13T23:57:14.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.076+0000 7fbf9742b640 1 -- 172.21.15.124:0/1898840574 wait complete. 2023-12-13T23:57:14.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 Processor -- start 2023-12-13T23:57:14.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 -- start start 2023-12-13T23:57:14.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf90143490 0x7fbf9012d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 0x7fbf9013fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90140b80 con 0x7fbf9015a4e0 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900ad180 con 0x7fbf9015e730 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf9742b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf900ad460 con 0x7fbf90143490 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf949c6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf949c6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45640/0 (socket says 172.21.15.124:45640) 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.078+0000 7fbf949c6640 1 -- 172.21.15.124:0/2535404886 learned_addr learned my addr 172.21.15.124:0/2535404886 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf959c8640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 0x7fbf9013fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf951c7640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf90143490 0x7fbf9012d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 -- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf90143490 msgr2=0x7fbf9012d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf90143490 0x7fbf9012d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 -- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 msgr2=0x7fbf9013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 0x7fbf9013fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 -- 172.21.15.124:0/2535404886 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf8c00b770 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf959c8640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 0x7fbf9013fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf949c6640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000c7f0 tx=0x7fbf8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80007820 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf80007980 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf80007cf0 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf900ad740 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.079+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf900ab5f0 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.080+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf90069400 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.080+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbf800113f0 con 0x7fbf9015a4e0 2023-12-13T23:57:14.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.081+0000 7fbf85ffb640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 0x7fbf7406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.081+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf80089930 con 0x7fbf9015a4e0 2023-12-13T23:57:14.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.081+0000 7fbf951c7640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 0x7fbf7406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.081+0000 7fbf951c7640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 0x7fbf7406d6b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c038000 tx=0x7fbf8c032000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.085+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbf800549b0 con 0x7fbf9015a4e0 2023-12-13T23:57:14.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.314+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbf90142e60 con 0x7fbf9015a4e0 2023-12-13T23:57:14.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.314+0000 7fbf85ffb640 1 -- 172.21.15.124:0/2535404886 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5298 (secure 0 0 0) 0x7fbf8001f090 con 0x7fbf9015a4e0 2023-12-13T23:57:14.313 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:14.313 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:14.313 INFO:teuthology.orchestra.run.smithi124.stdout: "auth_dump": [ 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.a", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplFo6KDBAArJJKNRTjxYdaYwtbHVbu7g==", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.b", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpl893rGBAAas73Q+Mux4HJCWgSronQlQ==", 2023-12-13T23:57:14.314 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.c", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpllD7vExAAj4trq2lGnNXIIi2G5k+/IQ==", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.315 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mds.d", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpliKtJIBAAz3MbEgTL5bv/l2zLj32n5w==", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.0", 2023-12-13T23:57:14.316 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAqRHplcCLWBxAAxYxXKFh6L9WK7uOg5UqRLA==", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.1", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAtRHplDW73BhAAQoBtr86nfF318aMqfI/7tA==", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.317 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.2", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAwRHpl+Bi6BBAAR9+uNWYlp4BCI3gmEbvp3Q==", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.3", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAzRHplrYhsAhAADvJtD6bIDoXVG50rEYkYaQ==", 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.318 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.4", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA5RHpleClsCxAAeVn2LqqB2gIUnF3HAimTnw==", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.319 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.5", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA8RHpl552xCRAAgCBrpQroUXTlg+wox8Fx/w==", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.6", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQA/RHplFgN+BxAAkUw1AQX9IJpOvNgPFBkQ3g==", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.320 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "osd.7", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQBCRHplw4NRBRAAbPuwtmaK8/9lrd4crZHHlg==", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow profile osd", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile osd", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.0", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw==", 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.321 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow r", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rw pool=cephfs_data" 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.1", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplz1vULRAANGEKQl90MW0+VmH0N96EXw==", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.322 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.2", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplGAlqKhAA3u5mk3DyH7KsWsAScLW0Ew==", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.3", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHpl7tuiMxAAF4ySYml4t9rxEC9Ji3/wdw==", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow", 2023-12-13T23:57:14.323 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow r", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow rw", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow rwx" 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "client.admin", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAlRHpl0JrlKxAAZUPeNVBB/yhD505NEeuUow==", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mgr": "allow *", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow *", 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.324 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mgr.x", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplEAcvCRAAuDLwmskaKHqzdX0/vTGodQ==", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile mgr", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "entity": "mgr.y", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "key": "AQAmRHplDCGRBRAATplO/LAqiqpuOu7VOMvxog==", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "caps": { 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mds": "allow *", 2023-12-13T23:57:14.325 INFO:teuthology.orchestra.run.smithi124.stdout: "mon": "allow profile mgr", 2023-12-13T23:57:14.326 INFO:teuthology.orchestra.run.smithi124.stdout: "osd": "allow *" 2023-12-13T23:57:14.326 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.326 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:14.326 INFO:teuthology.orchestra.run.smithi124.stdout: ] 2023-12-13T23:57:14.326 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 msgr2=0x7fbf7406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 0x7fbf7406d6b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c038000 tx=0x7fbf8c032000 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 msgr2=0x7fbf900ab0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fbf8000c7f0 tx=0x7fbf8000ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 shutdown_connections 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbf7406b1f0 0x7fbf7406d6b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbf9015e730 0x7fbf9013fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbf9015a4e0 0x7fbf900ab0b0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 --2- 172.21.15.124:0/2535404886 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbf90143490 0x7fbf9012d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 >> 172.21.15.124:0/2535404886 conn(0x7fbf9006a4d0 msgr2=0x7fbf900cd7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 shutdown_connections 2023-12-13T23:57:14.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.316+0000 7fbf9742b640 1 -- 172.21.15.124:0/2535404886 wait complete. 2023-12-13T23:57:14.329 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:14.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.430+0000 7fc5be37c640 1 Processor -- start 2023-12-13T23:57:14.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.430+0000 7fc5be37c640 1 -- start start 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5be37c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5b80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5be37c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b815e2a0 con 0x7fc5b815e730 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5be37c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b815b240 con 0x7fc5b815abf0 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5be37c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b815b520 con 0x7fc5b8143490 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b77fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc5b815e730 0x7fc5b815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47594/0 (socket says 172.21.15.124:47594) 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b77fe640 1 -- 172.21.15.124:0/1989574027 learned_addr learned my addr 172.21.15.124:0/1989574027 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5b80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2140812022 0 0) 0x7fc5b815e2a0 con 0x7fc5b815e730 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5a4003520 con 0x7fc5b815e730 2023-12-13T23:57:14.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390032167 0 0) 0x7fc5b815b240 con 0x7fc5b815abf0 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b815e2a0 con 0x7fc5b815abf0 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3425712891 0 0) 0x7fc5a4003520 con 0x7fc5b815e730 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 msgr2=0x7fc5b80b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5b80b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 >> v1:172.21.15.203:6789/0 conn(0x7fc5b815abf0 legacy=0x7fc5b80b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.431+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b815dc50 con 0x7fc5b815e730 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fc5a8001670 con 0x7fc5b815e730 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 >> v1:172.21.15.124:6789/0 conn(0x7fc5b815e730 legacy=0x7fc5b815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5a4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5a4004d00 0x7fc5a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b815b520 con 0x7fc5a4003b30 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a400dc40 con 0x7fc5a4004d00 2023-12-13T23:57:14.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a400de00 con 0x7fc5b8143490 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5a4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b77fe640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5bc919640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5a4004d00 0x7fc5a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 -- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 msgr2=0x7fc5a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b8143490 0x7fc5a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 -- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5a4004d00 msgr2=0x7fc5a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5a4004d00 0x7fc5a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.432+0000 7fc5b7fff640 1 -- 172.21.15.124:0/1989574027 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a4003520 con 0x7fc5a4003b30 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5a4003f30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009ac0 tx=0x7fc5a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f795934b1b9fc4c6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a002c6b0 con 0x7fc5a4003b30 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a002c810 con 0x7fc5a4003b30 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5b6ffd640 1 -- 172.21.15.124:0/1989574027 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a002cba0 con 0x7fc5a4003b30 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 -- 172.21.15.124:0/1989574027 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 msgr2=0x7fc5a4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 --2- 172.21.15.124:0/1989574027 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5a4003f30 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0009ac0 tx=0x7fc5a40057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5bc919640 1 -- 172.21.15.124:0/1989574027 reap_dead start 2023-12-13T23:57:14.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 -- 172.21.15.124:0/1989574027 shutdown_connections 2023-12-13T23:57:14.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 -- 172.21.15.124:0/1989574027 >> 172.21.15.124:0/1989574027 conn(0x7fc5b806a4d0 msgr2=0x7fc5b806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 -- 172.21.15.124:0/1989574027 shutdown_connections 2023-12-13T23:57:14.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.433+0000 7fc5be37c640 1 -- 172.21.15.124:0/1989574027 wait complete. 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 Processor -- start 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 -- start start 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b81eea30 0x7fc5b81eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5b812f750 0x7fc5b812fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80a84d0 con 0x7fc5a4003b30 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80a94e0 con 0x7fc5b812f750 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.435+0000 7fc5be37c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b80a8720 con 0x7fc5b81eea30 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45664/0 (socket says 172.21.15.124:45664) 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 -- 172.21.15.124:0/2986175226 learned_addr learned my addr 172.21.15.124:0/2986175226 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 -- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b81eea30 msgr2=0x7fc5b81eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b77fe640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b81eea30 0x7fc5b81eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5bc919640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5b812f750 0x7fc5b812fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b81eea30 0x7fc5b81eeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 -- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5b812f750 msgr2=0x7fc5b812fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5b812f750 0x7fc5b812fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 -- 172.21.15.124:0/2986175226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b80a8a00 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b7fff640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0000c00 tx=0x7fc5a0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0003810 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a0003970 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5b80a8ce0 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a0003d20 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.436+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b81ef3f0 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.437+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b8069400 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.438+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc5a0047450 con 0x7fc5a4003b30 2023-12-13T23:57:14.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.438+0000 7fc5b4ff9640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 0x7fc59806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.438+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc5a00ab8e0 con 0x7fc5a4003b30 2023-12-13T23:57:14.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.438+0000 7fc5b77fe640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 0x7fc59806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.439+0000 7fc5b77fe640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 0x7fc59806d6b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc5b81417f0 tx=0x7fc5a8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.441 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.442+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc5a0076bc0 con 0x7fc5a4003b30 2023-12-13T23:57:14.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.612+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b80a84d0 con 0x7fc5a4003b30 2023-12-13T23:57:14.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.613+0000 7fc5b4ff9640 1 -- 172.21.15.124:0/2986175226 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+12918 (secure 0 0 0) 0x7fc5a00044e0 con 0x7fc5a4003b30 2023-12-13T23:57:14.612 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:14.612 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":46,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:13.968346+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 msgr2=0x7fc59806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 0x7fc59806d6b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc5b81417f0 tx=0x7fc5a8009040 comp rx=0 tx=0).stop 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 msgr2=0x7fc5b8071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fc5a0000c00 tx=0x7fc5a0004730 comp rx=0 tx=0).stop 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 shutdown_connections 2023-12-13T23:57:14.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc59806b1f0 0x7fc59806d6b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5b812f750 0x7fc5b812fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc5b81eea30 0x7fc5b81eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 --2- 172.21.15.124:0/2986175226 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5a4003b30 0x7fc5b8071550 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 >> 172.21.15.124:0/2986175226 conn(0x7fc5b806a4d0 msgr2=0x7fc5b80b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 shutdown_connections 2023-12-13T23:57:14.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.615+0000 7fc5be37c640 1 -- 172.21.15.124:0/2986175226 wait complete. 2023-12-13T23:57:14.625 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:14.625 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:14.735 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.732+0000 7ff42b1c9640 1 Processor -- start 2023-12-13T23:57:14.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.732+0000 7ff42b1c9640 1 -- start start 2023-12-13T23:57:14.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff42b1c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff424144980 0x7ff42415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff42b1c9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff4240b4370 0x7ff4240b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff42b1c9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff424071770 con 0x7ff4240ceb70 2023-12-13T23:57:14.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff42b1c9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4240ccb60 con 0x7ff4240b4370 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff42b1c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4240b4ce0 con 0x7ff424144980 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff4240b4370 0x7ff4240b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff423fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ff4240ceb70 0x7ff4240cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47598/0 (socket says 172.21.15.124:47598) 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff423fff640 1 -- 172.21.15.124:0/2249405182 learned_addr learned my addr 172.21.15.124:0/2249405182 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff4240b4370 0x7ff4240b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45666/0 (socket says 172.21.15.124:45666) 2023-12-13T23:57:14.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 -- 172.21.15.124:0/2249405182 >> v2:172.21.15.203:3300/0 conn(0x7ff424144980 msgr2=0x7ff42415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 --2- 172.21.15.124:0/2249405182 >> v2:172.21.15.203:3300/0 conn(0x7ff424144980 0x7ff42415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 -- 172.21.15.124:0/2249405182 >> v1:172.21.15.124:6789/0 conn(0x7ff4240ceb70 legacy=0x7ff4240cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff4237fe640 1 -- 172.21.15.124:0/2249405182 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033527683 0 0) 0x7ff424071770 con 0x7ff4240ceb70 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.733+0000 7ff429766640 1 -- 172.21.15.124:0/2249405182 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240ccdc0 con 0x7ff4240b4370 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.734+0000 7ff429766640 1 --2- 172.21.15.124:0/2249405182 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b47a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff4240b4ec0 tx=0x7ff41c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bff292969d6fdc server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.734+0000 7ff4237fe640 1 -- 172.21.15.124:0/2249405182 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c02cc80 con 0x7ff4240b4370 2023-12-13T23:57:14.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.734+0000 7ff4237fe640 1 -- 172.21.15.124:0/2249405182 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c02cde0 con 0x7ff4240b4370 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.734+0000 7ff4237fe640 1 -- 172.21.15.124:0/2249405182 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff41c034660 con 0x7ff4240b4370 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 -- 172.21.15.124:0/2249405182 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 msgr2=0x7ff4240b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/2249405182 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b47a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7ff4240b4ec0 tx=0x7ff41c02bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 -- 172.21.15.124:0/2249405182 shutdown_connections 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/2249405182 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b47a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/2249405182 >> v2:172.21.15.203:3300/0 conn(0x7ff424144980 0x7ff42415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 -- 172.21.15.124:0/2249405182 >> 172.21.15.124:0/2249405182 conn(0x7ff42406a4d0 msgr2=0x7ff42406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 -- 172.21.15.124:0/2249405182 shutdown_connections 2023-12-13T23:57:14.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.735+0000 7ff42b1c9640 1 -- 172.21.15.124:0/2249405182 wait complete. 2023-12-13T23:57:14.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.736+0000 7ff42b1c9640 1 Processor -- start 2023-12-13T23:57:14.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.736+0000 7ff42b1c9640 1 -- start start 2023-12-13T23:57:14.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4240ceb70 0x7ff4240b3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff424144980 0x7ff4240b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240ad880 con 0x7ff4240b4370 2023-12-13T23:57:14.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240ad5a0 con 0x7ff424144980 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff42b1c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4240b2200 con 0x7ff4240ceb70 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45682/0 (socket says 172.21.15.124:45682) 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 -- 172.21.15.124:0/6550050 learned_addr learned my addr 172.21.15.124:0/6550050 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 -- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4240ceb70 msgr2=0x7ff4240b3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.747 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4240ceb70 0x7ff4240b3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 -- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff424144980 msgr2=0x7ff4240b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff429766640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff424144980 0x7ff4240b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff424144980 0x7ff4240b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 -- 172.21.15.124:0/6550050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c009a70 con 0x7ff4240b4370 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff428f65640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff41800b410 tx=0x7ff41800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.737+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418011830 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.738+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4180119b0 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.738+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4240b24e0 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.738+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff418011d60 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.738+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4240b3220 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.738+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4240b2a00 con 0x7ff4240b4370 2023-12-13T23:57:14.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.739+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff41801a400 con 0x7ff4240b4370 2023-12-13T23:57:14.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.739+0000 7ff4217fa640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 0x7ff40806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:14.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.740+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff418089860 con 0x7ff4240b4370 2023-12-13T23:57:14.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.740+0000 7ff423fff640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 0x7ff40806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:14.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.740+0000 7ff423fff640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 0x7ff40806d680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff410007fd0 tx=0x7ff410007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:14.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.744+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff4180548e0 con 0x7ff4240b4370 2023-12-13T23:57:14.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.908+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff424069400 con 0x7ff4240b4370 2023-12-13T23:57:14.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.908+0000 7ff4217fa640 1 -- 172.21.15.124:0/6550050 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff418015020 con 0x7ff4240b4370 2023-12-13T23:57:14.908 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:14.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 msgr2=0x7ff40806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 0x7ff40806d680 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff410007fd0 tx=0x7ff410007920 comp rx=0 tx=0).stop 2023-12-13T23:57:14.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 msgr2=0x7ff4240b2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:14.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff41800b410 tx=0x7ff41800b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 shutdown_connections 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff40806b1c0 0x7ff40806d680 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff424144980 0x7ff4240b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff4240ceb70 0x7ff4240b3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 --2- 172.21.15.124:0/6550050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff4240b4370 0x7ff4240b2ce0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 >> 172.21.15.124:0/6550050 conn(0x7ff42406a4d0 msgr2=0x7ff4241591b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 shutdown_connections 2023-12-13T23:57:14.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:14.910+0000 7ff42b1c9640 1 -- 172.21.15.124:0/6550050 wait complete. 2023-12-13T23:57:14.920 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-12-13T23:57:15.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.025+0000 7fc24164b640 1 Processor -- start 2023-12-13T23:57:15.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.025+0000 7fc24164b640 1 -- start start 2023-12-13T23:57:15.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc24164b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.028 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc24164b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc23c0ce150 0x7fc23c0b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc24164b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c0d1bb0 con 0x7fc23c0d2040 2023-12-13T23:57:15.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc24164b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c15d6b0 con 0x7fc23c0ce150 2023-12-13T23:57:15.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc24164b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c0ce700 con 0x7fc23c15d2d0 2023-12-13T23:57:15.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23affd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.029 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc23c0ce150 0x7fc23c0b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc23c0ce150 0x7fc23c0b51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:49860/0 (socket says 172.21.15.124:49860) 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23affd640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:43616/0 (socket says 172.21.15.124:43616) 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc233fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc23c0d2040 0x7fc23c0cda40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:34124/0 (socket says 172.21.15.124:34124) 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 -- 172.21.15.124:0/1737191434 learned_addr learned my addr 172.21.15.124:0/1737191434 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 -- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 msgr2=0x7fc23c0b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.030 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 -- 172.21.15.124:0/1737191434 >> v1:172.21.15.203:6789/0 conn(0x7fc23c0d2040 legacy=0x7fc23c0cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23b7fe640 1 -- 172.21.15.124:0/1737191434 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c0d1280 con 0x7fc23c0ce150 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.026+0000 7fc23affd640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.027+0000 7fc23b7fe640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0ce150 0x7fc23c0b51a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc234009e50 tx=0x7fc23402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2e7f6c00a54ebea server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.027+0000 7fc23a7fc640 1 -- 172.21.15.124:0/1737191434 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc23402cb00 con 0x7fc23c0ce150 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.027+0000 7fc23a7fc640 1 -- 172.21.15.124:0/1737191434 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc23402cc60 con 0x7fc23c0ce150 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.027+0000 7fc23a7fc640 1 -- 172.21.15.124:0/1737191434 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2340343f0 con 0x7fc23c0ce150 2023-12-13T23:57:15.031 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 -- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0ce150 msgr2=0x7fc23c0b51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0ce150 0x7fc23c0b51a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc234009e50 tx=0x7fc23402bf60 comp rx=0 tx=0).stop 2023-12-13T23:57:15.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 -- 172.21.15.124:0/1737191434 shutdown_connections 2023-12-13T23:57:15.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0ce150 0x7fc23c0b51a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.032 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 --2- 172.21.15.124:0/1737191434 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c0b4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 -- 172.21.15.124:0/1737191434 >> 172.21.15.124:0/1737191434 conn(0x7fc23c072050 msgr2=0x7fc23c072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:15.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 -- 172.21.15.124:0/1737191434 shutdown_connections 2023-12-13T23:57:15.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.028+0000 7fc24164b640 1 -- 172.21.15.124:0/1737191434 wait complete. 2023-12-13T23:57:15.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.029+0000 7fc24164b640 1 Processor -- start 2023-12-13T23:57:15.033 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.029+0000 7fc24164b640 1 -- start start 2023-12-13T23:57:15.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0d2040 0x7fc23c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c15e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f6a30 con 0x7fc23c0ce150 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f6d10 con 0x7fc23c0d2040 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc24164b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f6ff0 con 0x7fc23c15d2d0 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45696/0 (socket says 172.21.15.124:45696) 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 -- 172.21.15.124:0/3630989310 learned_addr learned my addr 172.21.15.124:0/3630989310 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc233fff640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0d2040 0x7fc23c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23b7fe640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c15e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 -- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 msgr2=0x7fc23c15e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c15e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.043 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.030+0000 7fc23affd640 1 -- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0d2040 msgr2=0x7fc23c15f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc23affd640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0d2040 0x7fc23c15f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc23affd640 1 -- 172.21.15.124:0/3630989310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22c009400 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc23affd640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00e960 tx=0x7fc22c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c00ca00 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc22c00cb60 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc234009b30 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc22c013470 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.031+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c15db70 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.032+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c069400 con 0x7fc23c0ce150 2023-12-13T23:57:15.044 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.032+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc22c0135d0 con 0x7fc23c0ce150 2023-12-13T23:57:15.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.033+0000 7fc2337fe640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 0x7fc21c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.033+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc22c0898e0 con 0x7fc23c0ce150 2023-12-13T23:57:15.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.033+0000 7fc233fff640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 0x7fc21c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.033+0000 7fc233fff640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 0x7fc21c06d6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0725a0 tx=0x7fc224007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.045 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.037+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc22c054970 con 0x7fc23c0ce150 2023-12-13T23:57:15.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.205+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc23c15cc10 con 0x7fc23c0ce150 2023-12-13T23:57:15.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.206+0000 7fc2337fe640 1 -- 172.21.15.124:0/3630989310 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v46) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc22c057f90 con 0x7fc23c0ce150 2023-12-13T23:57:15.205 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:15.205 INFO:teuthology.orchestra.run.smithi124.stdout:[ 2023-12-13T23:57:15.205 INFO:teuthology.orchestra.run.smithi124.stdout: { 2023-12-13T23:57:15.206 INFO:teuthology.orchestra.run.smithi124.stdout: "poolnum": 1, 2023-12-13T23:57:15.206 INFO:teuthology.orchestra.run.smithi124.stdout: "poolname": ".mgr" 2023-12-13T23:57:15.206 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:15.206 INFO:teuthology.orchestra.run.smithi124.stdout:] 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.207+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 msgr2=0x7fc21c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.207+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 0x7fc21c06d6b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc23c0725a0 tx=0x7fc224007ca0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.207+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 msgr2=0x7fc23c1f6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.207+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00e960 tx=0x7fc22c00ee20 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 shutdown_connections 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc21c06b1f0 0x7fc21c06d6b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc23c0d2040 0x7fc23c15f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc23c15d2d0 0x7fc23c15e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 --2- 172.21.15.124:0/3630989310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc23c0ce150 0x7fc23c1f6100 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 >> 172.21.15.124:0/3630989310 conn(0x7fc23c072050 msgr2=0x7fc23c0b3700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:15.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 shutdown_connections 2023-12-13T23:57:15.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.208+0000 7fc24164b640 1 -- 172.21.15.124:0/3630989310 wait complete. 2023-12-13T23:57:15.218 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-12-13T23:57:15.218 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-12-13T23:57:15.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.321+0000 7f1978ad0640 1 Processor -- start 2023-12-13T23:57:15.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.321+0000 7f1978ad0640 1 -- start start 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1974143490 0x7f19740b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f197415e730 0x7f19740b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f197415a4e0 0x7f197415a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f197415b520 con 0x7f1974143490 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1974143870 con 0x7f197415e730 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.322+0000 7f1978ad0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f197415e2a0 con 0x7f197415a4e0 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1974143490 0x7f19740b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1974143490 0x7f19740b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45700/0 (socket says 172.21.15.124:45700) 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 -- 172.21.15.124:0/2292658357 learned_addr learned my addr 172.21.15.124:0/2292658357 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:15.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f1971d9b640 1 --2- 172.21.15.124:0/2292658357 >> v2:172.21.15.203:3300/0 conn(0x7f197415e730 0x7f19740b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 -- 172.21.15.124:0/2292658357 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f197415a4e0 msgr2=0x7f197415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 --2- 172.21.15.124:0/2292658357 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f197415a4e0 0x7f197415a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 -- 172.21.15.124:0/2292658357 >> v2:172.21.15.203:3300/0 conn(0x7f197415e730 msgr2=0x7f19740b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 --2- 172.21.15.124:0/2292658357 >> v2:172.21.15.203:3300/0 conn(0x7f197415e730 0x7f19740b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.323+0000 7f197259c640 1 -- 172.21.15.124:0/2292658357 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f197415b240 con 0x7f1974143490 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f197259c640 1 --2- 172.21.15.124:0/2292658357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1974143490 0x7f19740b0cf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f19740aeca0 tx=0x7f196802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffed0a4046e591 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f197159a640 1 -- 172.21.15.124:0/2292658357 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968031870 con 0x7f1974143490 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f197159a640 1 -- 172.21.15.124:0/2292658357 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19680319d0 con 0x7f1974143490 2023-12-13T23:57:15.324 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f197159a640 1 -- 172.21.15.124:0/2292658357 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1968031d60 con 0x7f1974143490 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 -- 172.21.15.124:0/2292658357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1974143490 msgr2=0x7f19740b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 --2- 172.21.15.124:0/2292658357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1974143490 0x7f19740b0cf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f19740aeca0 tx=0x7f196802b910 comp rx=0 tx=0).stop 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 -- 172.21.15.124:0/2292658357 shutdown_connections 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 --2- 172.21.15.124:0/2292658357 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f197415a4e0 0x7f197415a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 --2- 172.21.15.124:0/2292658357 >> v2:172.21.15.203:3300/0 conn(0x7f197415e730 0x7f19740b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 --2- 172.21.15.124:0/2292658357 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1974143490 0x7f19740b0cf0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 -- 172.21.15.124:0/2292658357 >> 172.21.15.124:0/2292658357 conn(0x7f197406a4d0 msgr2=0x7f197406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:15.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 -- 172.21.15.124:0/2292658357 shutdown_connections 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.324+0000 7f1978ad0640 1 -- 172.21.15.124:0/2292658357 wait complete. 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 Processor -- start 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 -- start start 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 0x7f19740abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f197415e730 0x7f19740adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19740ac2e0 con 0x7f197415a4e0 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1974130b20 con 0x7f197415e730 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1978ad0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1974130e00 con 0x7f1974143490 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.326+0000 7f1971d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45714/0 (socket says 172.21.15.124:45714) 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 -- 172.21.15.124:0/4226392329 learned_addr learned my addr 172.21.15.124:0/4226392329 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f197259c640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 0x7f19740abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.328 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1972d9d640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f197415e730 0x7f19740adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 -- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 msgr2=0x7f19740abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 0x7f19740abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 -- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f197415e730 msgr2=0x7f19740adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f197415e730 0x7f19740adfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 -- 172.21.15.124:0/4226392329 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19680094d0 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f197259c640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 0x7f19740abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1971d9b640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f196400c7f0 tx=0x7f196400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964004220 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1964004380 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19741310e0 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964007840 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.327+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19741316a0 con 0x7f197415a4e0 2023-12-13T23:57:15.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.328+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19740b4380 con 0x7f197415a4e0 2023-12-13T23:57:15.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.328+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1964007bb0 con 0x7f197415a4e0 2023-12-13T23:57:15.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.329+0000 7f1962ffd640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 0x7f195406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:15.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.329+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f19640896a0 con 0x7f197415a4e0 2023-12-13T23:57:15.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.329+0000 7f197259c640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 0x7f195406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:15.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.330+0000 7f197259c640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 0x7f195406d680 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1968036000 tx=0x7f196803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:15.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.333+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f19640546a0 con 0x7f197415a4e0 2023-12-13T23:57:15.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.498+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f197412e7b0 con 0x7f197415a4e0 2023-12-13T23:57:15.975 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.976+0000 7f1962ffd640 1 -- 172.21.15.124:0/4226392329 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7f197412e7b0 con 0x7f197415a4e0 2023-12-13T23:57:15.975 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' created 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 msgr2=0x7f195406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 0x7f195406d680 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f1968036000 tx=0x7f196803c040 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 msgr2=0x7f19740ada70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f196400c7f0 tx=0x7f196400ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 shutdown_connections 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f195406b1c0 0x7f195406d680 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f197415e730 0x7f19740adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f197415a4e0 0x7f19740ada70 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 --2- 172.21.15.124:0/4226392329 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1974143490 0x7f19740abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 >> 172.21.15.124:0/4226392329 conn(0x7f197406a4d0 msgr2=0x7f19740b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.978+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 shutdown_connections 2023-12-13T23:57:15.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:15.979+0000 7f1978ad0640 1 -- 172.21.15.124:0/4226392329 wait complete. 2023-12-13T23:57:15.993 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-12-13T23:57:16.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 Processor -- start 2023-12-13T23:57:16.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 -- start start 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7effc40cec80 0x7effc406b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effc40cf060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc40ce9e0 con 0x7effc40d0d10 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc406bb20 con 0x7effc40cec80 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.104+0000 7effc9ba1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40d0510 con 0x7effc406cff0 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc93a0640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7effc40d0d10 0x7effc40d11a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47604/0 (socket says 172.21.15.124:47604) 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc93a0640 1 -- 172.21.15.124:0/976246242 learned_addr learned my addr 172.21.15.124:0/976246242 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172316971 0 0) 0x7effbc001490 con 0x7effc40d0d10 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb0003520 con 0x7effc40d0d10 2023-12-13T23:57:16.108 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc3fff640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effc40cf060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc8b9f640 1 --2- 172.21.15.124:0/976246242 >> v2:172.21.15.203:3300/0 conn(0x7effc40cec80 0x7effc406b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3061856564 0 0) 0x7effb0003520 con 0x7effc40d0d10 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 msgr2=0x7effc40cf060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effc40cf060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 >> v2:172.21.15.203:3300/0 conn(0x7effc40cec80 msgr2=0x7effc406b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 --2- 172.21.15.124:0/976246242 >> v2:172.21.15.203:3300/0 conn(0x7effc40cec80 0x7effc406b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40ce550 con 0x7effc40d0d10 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7effbc001b80 con 0x7effc40d0d10 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 >> v1:172.21.15.124:6789/0 conn(0x7effc40d0d10 legacy=0x7effc40d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb0003970 0x7effb0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effb00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb0004d00 0x7effb0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc406bb20 con 0x7effb0004d00 2023-12-13T23:57:16.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40d0510 con 0x7effb0003970 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb000ba00 con 0x7effc406cff0 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc93a0640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb0004d00 0x7effb0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc3fff640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effb00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.105+0000 7effc8b9f640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb0003970 0x7effb0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 -- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 msgr2=0x7effb00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effb00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 -- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb0003970 msgr2=0x7effb0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb0003970 0x7effb0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 -- 172.21.15.124:0/976246242 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb0003520 con 0x7effb0004d00 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc3fff640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc406cff0 0x7effb00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc8b9f640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effb0003970 0x7effb0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc93a0640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb0004d00 0x7effb0005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effbc00a830 tx=0x7effb00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e23666d73ad435 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.106+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc034630 con 0x7effb0004d00 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc02f070 con 0x7effb0004d00 2023-12-13T23:57:16.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc37fe640 1 -- 172.21.15.124:0/976246242 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc0349d0 con 0x7effb0004d00 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 -- 172.21.15.124:0/976246242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb0004d00 msgr2=0x7effb0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 --2- 172.21.15.124:0/976246242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effb0004d00 0x7effb0005180 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effbc00a830 tx=0x7effb00057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc93a0640 1 -- 172.21.15.124:0/976246242 reap_dead start 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 -- 172.21.15.124:0/976246242 shutdown_connections 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 -- 172.21.15.124:0/976246242 >> 172.21.15.124:0/976246242 conn(0x7effc4099a70 msgr2=0x7effc4099e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 -- 172.21.15.124:0/976246242 shutdown_connections 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.107+0000 7effc9ba1640 1 -- 172.21.15.124:0/976246242 wait complete. 2023-12-13T23:57:16.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.108+0000 7effc9ba1640 1 Processor -- start 2023-12-13T23:57:16.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.108+0000 7effc9ba1640 1 -- start start 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc40b0f00 0x7effc40cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc40cbdf0 0x7effc40c9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40b1360 con 0x7effc414b710 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc40adbd0 con 0x7effc40cbdf0 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc9ba1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc40cc270 con 0x7effc40b0f00 2023-12-13T23:57:16.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45736/0 (socket says 172.21.15.124:45736) 2023-12-13T23:57:16.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 -- 172.21.15.124:0/684655441 learned_addr learned my addr 172.21.15.124:0/684655441 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:16.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 -- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc40b0f00 msgr2=0x7effc40cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc40b0f00 0x7effc40cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 -- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc40cbdf0 msgr2=0x7effc40c9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.109+0000 7effc8b9f640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc40cbdf0 0x7effc40c9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc8b9f640 1 -- 172.21.15.124:0/684655441 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40ade20 con 0x7effc414b710 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc8b9f640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effac0115d0 tx=0x7effac011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac015870 con 0x7effc414b710 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac0159d0 con 0x7effc414b710 2023-12-13T23:57:16.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc40ae100 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac015d60 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.110+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc40ae6c0 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.111+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc404e8b0 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.111+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7effac00f460 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.112+0000 7effc17fa640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 0x7effa406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.112+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5227+0+0 (secure 0 0 0) 0x7effac08a050 con 0x7effc414b710 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.112+0000 7effc3fff640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 0x7effa406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:16.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.113+0000 7effc3fff640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 0x7effa406d6b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7effc40aeff0 tx=0x7effb4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:16.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.116+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7effac054ed0 con 0x7effc414b710 2023-12-13T23:57:16.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.284+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7effc406c610 con 0x7effc414b710 2023-12-13T23:57:16.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.980+0000 7effc17fa640 1 -- 172.21.15.124:0/684655441 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v48) v1 ==== 170+0+0 (secure 0 0 0) 0x7effac0584f0 con 0x7effc414b710 2023-12-13T23:57:16.980 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' created 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 msgr2=0x7effa406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 0x7effa406d6b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7effc40aeff0 tx=0x7effb4008040 comp rx=0 tx=0).stop 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 msgr2=0x7effc40b09c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effac0115d0 tx=0x7effac011a90 comp rx=0 tx=0).stop 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 shutdown_connections 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7effa406b1f0 0x7effa406d6b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7effc40cbdf0 0x7effc40c9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7effc40b0f00 0x7effc40cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 --2- 172.21.15.124:0/684655441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc414b710 0x7effc40b09c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:16.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 >> 172.21.15.124:0/684655441 conn(0x7effc4099a70 msgr2=0x7effc406e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:16.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 shutdown_connections 2023-12-13T23:57:16.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:16.983+0000 7effc9ba1640 1 -- 172.21.15.124:0/684655441 wait complete. 2023-12-13T23:57:17.005 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-12-13T23:57:17.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.110+0000 7fac9fada640 1 Processor -- start 2023-12-13T23:57:17.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 -- start start 2023-12-13T23:57:17.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 0x7fac980b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980b2b10 con 0x7fac980d2910 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac980d1590 con 0x7fac980d2d90 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.111+0000 7fac9fada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980d1870 con 0x7fac9815d330 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9e077640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d876640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 0x7fac980b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d075640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fac980d2910 0x7fac980d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:34140/0 (socket says 172.21.15.124:34140) 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d075640 1 -- 172.21.15.124:0/1100839717 learned_addr learned my addr 172.21.15.124:0/1100839717 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9e077640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:49938/0 (socket says 172.21.15.124:49938) 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d876640 1 -- 172.21.15.124:0/1100839717 >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 msgr2=0x7fac980b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d876640 1 --2- 172.21.15.124:0/1100839717 >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d876640 1 -- 172.21.15.124:0/1100839717 >> v1:172.21.15.203:6789/0 conn(0x7fac980d2910 legacy=0x7fac980d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9d876640 1 -- 172.21.15.124:0/1100839717 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac980d1b50 con 0x7fac9815d330 2023-12-13T23:57:17.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.112+0000 7fac9e077640 1 --2- 172.21.15.124:0/1100839717 >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9d876640 1 --2- 172.21.15.124:0/1100839717 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 0x7fac980b8b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fac880097a0 tx=0x7fac8802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd6f1198181eef16 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9c874640 1 -- 172.21.15.124:0/1100839717 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8802e070 con 0x7fac9815d330 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9c874640 1 -- 172.21.15.124:0/1100839717 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac880315a0 con 0x7fac9815d330 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9c874640 1 -- 172.21.15.124:0/1100839717 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88031930 con 0x7fac9815d330 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9fada640 1 -- 172.21.15.124:0/1100839717 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 msgr2=0x7fac980b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.113+0000 7fac9fada640 1 --2- 172.21.15.124:0/1100839717 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 0x7fac980b8b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fac880097a0 tx=0x7fac8802b840 comp rx=0 tx=0).stop 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 -- 172.21.15.124:0/1100839717 shutdown_connections 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 --2- 172.21.15.124:0/1100839717 >> v2:172.21.15.203:3300/0 conn(0x7fac980d2d90 0x7fac980b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 --2- 172.21.15.124:0/1100839717 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac9815d330 0x7fac980b8b80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 -- 172.21.15.124:0/1100839717 >> 172.21.15.124:0/1100839717 conn(0x7fac98072020 msgr2=0x7fac98072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 -- 172.21.15.124:0/1100839717 shutdown_connections 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.114+0000 7fac9fada640 1 -- 172.21.15.124:0/1100839717 wait complete. 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 Processor -- start 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 -- start start 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac980d2d90 0x7fac98156340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac9815d330 0x7fac98153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980c9590 con 0x7fac980d2910 2023-12-13T23:57:17.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac980c9100 con 0x7fac9815d330 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.115+0000 7fac9fada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980c84f0 con 0x7fac980d2d90 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45748/0 (socket says 172.21.15.124:45748) 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 -- 172.21.15.124:0/1705423510 learned_addr learned my addr 172.21.15.124:0/1705423510 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d075640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac980d2d90 0x7fac98156340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9e077640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac9815d330 0x7fac98153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 -- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac980d2d90 msgr2=0x7fac98156340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac980d2d90 0x7fac98156340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 -- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac9815d330 msgr2=0x7fac98153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac9815d330 0x7fac98153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 -- 172.21.15.124:0/1705423510 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac90009a70 con 0x7fac980d2910 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9d876640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fac88034040 tx=0x7fac880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88032630 con 0x7fac980d2910 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac88032790 con 0x7fac980d2910 2023-12-13T23:57:17.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac88032b40 con 0x7fac980d2910 2023-12-13T23:57:17.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac88009400 con 0x7fac980d2910 2023-12-13T23:57:17.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.116+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac980c8ab0 con 0x7fac980d2910 2023-12-13T23:57:17.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.117+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac98069400 con 0x7fac980d2910 2023-12-13T23:57:17.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.118+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fac88032ca0 con 0x7fac980d2910 2023-12-13T23:57:17.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.118+0000 7fac8e7fc640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 0x7fac7c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:17.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.118+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fac880ab720 con 0x7fac980d2910 2023-12-13T23:57:17.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.118+0000 7fac9d075640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 0x7fac7c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:17.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.119+0000 7fac9d075640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 0x7fac7c06d6b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fac980b1f90 tx=0x7fac94007e80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:17.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.123+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fac88076540 con 0x7fac980d2910 2023-12-13T23:57:17.323 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.324+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fac980b8a90 con 0x7fac980d2910 2023-12-13T23:57:17.981 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.982+0000 7fac8e7fc640 1 -- 172.21.15.124:0/1705423510 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7fac88079b60 con 0x7fac980d2910 2023-12-13T23:57:17.981 INFO:teuthology.orchestra.run.smithi124.stderr:new fs with metadata pool 7 and data pool 8 2023-12-13T23:57:17.983 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 msgr2=0x7fac7c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 0x7fac7c06d6b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fac980b1f90 tx=0x7fac94007e80 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 msgr2=0x7fac980c7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fac88034040 tx=0x7fac880046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 shutdown_connections 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fac7c06b1f0 0x7fac7c06d6b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fac9815d330 0x7fac98153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fac980d2d90 0x7fac98156340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 --2- 172.21.15.124:0/1705423510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fac980d2910 0x7fac980c7bf0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 >> 172.21.15.124:0/1705423510 conn(0x7fac98072020 msgr2=0x7fac98072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 shutdown_connections 2023-12-13T23:57:17.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:17.984+0000 7fac9fada640 1 -- 172.21.15.124:0/1705423510 wait complete. 2023-12-13T23:57:18.001 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:18.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.106+0000 7f51011a0640 1 Processor -- start 2023-12-13T23:57:18.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.106+0000 7f51011a0640 1 -- start start 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f51011a0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f50fc161cc0 0x7f50fc168440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f51011a0640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc06eb50 con 0x7f50fc06fe60 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f51011a0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50fc06edc0 con 0x7f50fc161100 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f51011a0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50fc06f0a0 con 0x7f50fc161cc0 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f50fb7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f50fc06fe60 0x7f50fc06c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47620/0 (socket says 172.21.15.124:47620) 2023-12-13T23:57:18.109 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.107+0000 7f50fb7fe640 1 -- 172.21.15.124:0/256327982 learned_addr learned my addr 172.21.15.124:0/256327982 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102572778 0 0) 0x7f50fc06eb50 con 0x7f50fc06fe60 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50e4003520 con 0x7f50fc06fe60 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> v2:172.21.15.203:3300/0 conn(0x7f50fc161cc0 0x7f50fc168440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1193886207 0 0) 0x7f50e4003520 con 0x7f50fc06fe60 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 >> v2:172.21.15.203:3300/0 conn(0x7f50fc161cc0 msgr2=0x7f50fc168440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 --2- 172.21.15.124:0/256327982 >> v2:172.21.15.203:3300/0 conn(0x7f50fc161cc0 0x7f50fc168440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 >> v1:172.21.15.203:6789/0 conn(0x7f50fc161100 legacy=0x7f50fc1614e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc06f380 con 0x7f50fc06fe60 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f50f0001670 con 0x7f50fc06fe60 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> v2:172.21.15.203:3300/0 conn(0x7f50fc161cc0 0x7f50fc168440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 >> v1:172.21.15.124:6789/0 conn(0x7f50fc06fe60 legacy=0x7f50fc06c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e4003970 0x7f50e4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e40047c0 0x7f50e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.110 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc06f0a0 con 0x7f50e4005180 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e400c0b0 con 0x7f50e4003970 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e400c270 con 0x7f50e40047c0 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50fbfff640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e4003970 0x7f50e4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f50fb7fe640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e40047c0 0x7f50e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 -- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e40047c0 msgr2=0x7f50e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e40047c0 0x7f50e4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 -- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e4003970 msgr2=0x7f50e4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e4003970 0x7f50e4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.108+0000 7f510099f640 1 -- 172.21.15.124:0/256327982 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e4003520 con 0x7f50e4005180 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.109+0000 7f510099f640 1 -- 172.21.15.124:0/256327982 reap_dead start 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.109+0000 7f510099f640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50e400b980 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f50f40073e0 tx=0x7f50e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32bf63ea9bcb05 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.109+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f402cdc0 con 0x7f50e4005180 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.109+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f4034440 con 0x7f50e4005180 2023-12-13T23:57:18.111 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.109+0000 7f50faffd640 1 -- 172.21.15.124:0/256327982 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f40347d0 con 0x7f50e4005180 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 -- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 msgr2=0x7f50e400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50e400b980 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f50f40073e0 tx=0x7f50e40057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 -- 172.21.15.124:0/256327982 shutdown_connections 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 --2- 172.21.15.124:0/256327982 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50e400b980 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 -- 172.21.15.124:0/256327982 >> 172.21.15.124:0/256327982 conn(0x7f50fc0b28f0 msgr2=0x7f50fc0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 -- 172.21.15.124:0/256327982 shutdown_connections 2023-12-13T23:57:18.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.110+0000 7f51011a0640 1 -- 172.21.15.124:0/256327982 wait complete. 2023-12-13T23:57:18.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.111+0000 7f51011a0640 1 Processor -- start 2023-12-13T23:57:18.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.111+0000 7f51011a0640 1 -- start start 2023-12-13T23:57:18.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50fc0c71d0 0x7f50fc0b5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 0x7f50fc15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0c6db0 con 0x7f50e4005180 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc15b750 con 0x7f50fc15ad90 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f51011a0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc0c6130 con 0x7f50fc0c71d0 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45768/0 (socket says 172.21.15.124:45768) 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 -- 172.21.15.124:0/3588675725 learned_addr learned my addr 172.21.15.124:0/3588675725 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f510099f640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 0x7f50fc15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fb7fe640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50fc0c71d0 0x7f50fc0b5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 -- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50fc0c71d0 msgr2=0x7f50fc0b5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50fc0c71d0 0x7f50fc0b5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 -- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 msgr2=0x7f50fc15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 0x7f50fc15b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f50fbfff640 1 -- 172.21.15.124:0/3588675725 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc0c6410 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.112+0000 7f510099f640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 0x7f50fc15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f50fbfff640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00f620 tx=0x7f50ec00fae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec0052e0 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50ec005440 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50ec003810 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50fc0c66f0 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.113+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50fc0c7650 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.114+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50fc069320 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.114+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f50ec003970 con 0x7f50e4005180 2023-12-13T23:57:18.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.115+0000 7f50f8ff9640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 0x7f50e806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.115+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50ec0897f0 con 0x7f50e4005180 2023-12-13T23:57:18.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.115+0000 7f50fb7fe640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 0x7f50e806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.115+0000 7f50fb7fe640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 0x7f50e806d680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f50fc15f670 tx=0x7f50f0009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.119+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f50ec0545b0 con 0x7f50e4005180 2023-12-13T23:57:18.286 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.287+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50fc06b2c0 con 0x7f50e4005180 2023-12-13T23:57:18.287 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.288+0000 7f50f8ff9640 1 -- 172.21.15.124:0/3588675725 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16643 (secure 0 0 0) 0x7f50ec057bd0 con 0x7f50e4005180 2023-12-13T23:57:18.287 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:18.287 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":49,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:17.326160+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 msgr2=0x7f50e806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 0x7f50e806d680 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f50fc15f670 tx=0x7f50f0009040 comp rx=0 tx=0).stop 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 msgr2=0x7f50fc15f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f50ec00f620 tx=0x7f50ec00fae0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 shutdown_connections 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50e806b1c0 0x7f50e806d680 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.289 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50fc15ad90 0x7f50fc15b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50fc0c71d0 0x7f50fc0b5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 --2- 172.21.15.124:0/3588675725 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e4005180 0x7f50fc15f950 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 >> 172.21.15.124:0/3588675725 conn(0x7f50fc0b28f0 msgr2=0x7f50fc0b1ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:18.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 shutdown_connections 2023-12-13T23:57:18.290 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.290+0000 7f51011a0640 1 -- 172.21.15.124:0/3588675725 wait complete. 2023-12-13T23:57:18.300 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:18.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.401+0000 7eff2559e640 1 Processor -- start 2023-12-13T23:57:18.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.401+0000 7eff2559e640 1 -- start start 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff2559e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff200b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff2559e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7eff200d2040 0x7eff200b51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff2559e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff200d1bb0 con 0x7eff200cda40 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff2559e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2015d6b0 con 0x7eff200d2040 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff2559e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff200ce700 con 0x7eff2015d2d0 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1f7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7eff200cda40 0x7eff200cded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47630/0 (socket says 172.21.15.124:47630) 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1f7fe640 1 -- 172.21.15.124:0/3470987147 learned_addr learned my addr 172.21.15.124:0/3470987147 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:18.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1effd640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff200b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1e7fc640 1 --2- 172.21.15.124:0/3470987147 >> v2:172.21.15.203:3300/0 conn(0x7eff200d2040 0x7eff200b51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034491470 0 0) 0x7eff200d1bb0 con 0x7eff200cda40 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff08003520 con 0x7eff200cda40 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999537239 0 0) 0x7eff08003520 con 0x7eff200cda40 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 msgr2=0x7eff200b4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.402+0000 7eff1dffb640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff200b4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 >> v2:172.21.15.203:3300/0 conn(0x7eff200d2040 msgr2=0x7eff200b51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 --2- 172.21.15.124:0/3470987147 >> v2:172.21.15.203:3300/0 conn(0x7eff200d2040 0x7eff200b51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff200d1280 con 0x7eff200cda40 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7eff180019f0 con 0x7eff200cda40 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 >> v1:172.21.15.124:6789/0 conn(0x7eff200cda40 legacy=0x7eff200cded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff08003970 0x7eff08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff08004d00 0x7eff08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2015d6b0 con 0x7eff08003970 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200ce700 con 0x7eff08004d00 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0800b980 con 0x7eff2015d2d0 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff08003970 0x7eff08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1e7fc640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1f7fe640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff08004d00 0x7eff08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 -- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 msgr2=0x7eff080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff2015d2d0 0x7eff080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 -- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff08004d00 msgr2=0x7eff08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff08004d00 0x7eff08005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1effd640 1 -- 172.21.15.124:0/3470987147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff08003520 con 0x7eff08003970 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.403+0000 7eff1f7fe640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff08004d00 0x7eff08005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff1effd640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff08003970 0x7eff08003d90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7eff10009ac0 tx=0x7eff080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c9291904316e2b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1002c6b0 con 0x7eff08003970 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1002c810 con 0x7eff08003970 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff1dffb640 1 -- 172.21.15.124:0/3470987147 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1002cba0 con 0x7eff08003970 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 -- 172.21.15.124:0/3470987147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff08003970 msgr2=0x7eff08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 --2- 172.21.15.124:0/3470987147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff08003970 0x7eff08003d90 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7eff10009ac0 tx=0x7eff080057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff1f7fe640 1 -- 172.21.15.124:0/3470987147 reap_dead start 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 -- 172.21.15.124:0/3470987147 shutdown_connections 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 -- 172.21.15.124:0/3470987147 >> 172.21.15.124:0/3470987147 conn(0x7eff20072050 msgr2=0x7eff20072460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 -- 172.21.15.124:0/3470987147 shutdown_connections 2023-12-13T23:57:18.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.404+0000 7eff2559e640 1 -- 172.21.15.124:0/3470987147 wait complete. 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 Processor -- start 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 -- start start 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff08003970 0x7eff20159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 0x7eff200c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200caa10 con 0x7eff200cbab0 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff200cacf0 con 0x7eff200ca550 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff2559e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff200cafd0 con 0x7eff08003970 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.406+0000 7eff1f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45782/0 (socket says 172.21.15.124:45782) 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 -- 172.21.15.124:0/975105774 learned_addr learned my addr 172.21.15.124:0/975105774 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:18.410 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1effd640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff08003970 0x7eff20159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1e7fc640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 0x7eff200c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 -- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff08003970 msgr2=0x7eff20159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff08003970 0x7eff20159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 -- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 msgr2=0x7eff200c76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 0x7eff200c76b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 -- 172.21.15.124:0/975105774 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff200cb2b0 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1e7fc640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 0x7eff200c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff1f7fe640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7eff2006c5d0 tx=0x7eff1800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18007a80 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1800eda0 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff200cb590 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff18013660 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.407+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff200c7bf0 con 0x7eff200cbab0 2023-12-13T23:57:18.411 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.408+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff200cb780 con 0x7eff200cbab0 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.409+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7eff180137c0 con 0x7eff200cbab0 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.409+0000 7eff077fe640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 0x7eff0c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.409+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff18089e50 con 0x7eff200cbab0 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.409+0000 7eff1effd640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 0x7eff0c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.410+0000 7eff1effd640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 0x7eff0c06d6b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7eff10036000 tx=0x7eff10030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.412 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.413+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7eff18054b80 con 0x7eff200cbab0 2023-12-13T23:57:18.580 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.580+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff20069400 con 0x7eff200cbab0 2023-12-13T23:57:18.581 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.582+0000 7eff077fe640 1 -- 172.21.15.124:0/975105774 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v49) v1 ==== 74+0+16643 (secure 0 0 0) 0x7eff180581a0 con 0x7eff200cbab0 2023-12-13T23:57:18.581 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:18.581 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":49,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:17.326160+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 msgr2=0x7eff0c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 0x7eff0c06d6b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7eff10036000 tx=0x7eff10030000 comp rx=0 tx=0).stop 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 msgr2=0x7eff200b7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7eff2006c5d0 tx=0x7eff1800e530 comp rx=0 tx=0).stop 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 shutdown_connections 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7eff0c06b1f0 0x7eff0c06d6b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7eff200ca550 0x7eff200c76b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7eff200cbab0 0x7eff200b7280 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 --2- 172.21.15.124:0/975105774 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7eff08003970 0x7eff20159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 >> 172.21.15.124:0/975105774 conn(0x7eff20072050 msgr2=0x7eff2015d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:18.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 shutdown_connections 2023-12-13T23:57:18.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.584+0000 7eff2559e640 1 -- 172.21.15.124:0/975105774 wait complete. 2023-12-13T23:57:18.594 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-12-13T23:57:18.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.697+0000 7f02de475640 1 Processor -- start 2023-12-13T23:57:18.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.697+0000 7f02de475640 1 -- start start 2023-12-13T23:57:18.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.697+0000 7f02de475640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f02d806b4e0 0x7f02d806b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.697+0000 7f02de475640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02d80ce800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02de475640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d80ccc40 con 0x7f02d813d080 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02de475640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02d813d460 con 0x7f02d806b4e0 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02de475640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d806beb0 con 0x7f02d80ce3d0 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d7fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f02d813d080 0x7f02d815aff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47646/0 (socket says 172.21.15.124:47646) 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d7fff640 1 -- 172.21.15.124:0/1623629677 learned_addr learned my addr 172.21.15.124:0/1623629677 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785550558 0 0) 0x7f02d80ccc40 con 0x7f02d813d080 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c0003520 con 0x7f02d813d080 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d77fe640 1 --2- 172.21.15.124:0/1623629677 >> v2:172.21.15.203:3300/0 conn(0x7f02d806b4e0 0x7f02d806b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02dca12640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02d80ce800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 684477268 0 0) 0x7f02c0003520 con 0x7f02d813d080 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 msgr2=0x7f02d80ce800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02d80ce800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.700 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 >> v2:172.21.15.203:3300/0 conn(0x7f02d806b4e0 msgr2=0x7f02d806b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 --2- 172.21.15.124:0/1623629677 >> v2:172.21.15.203:3300/0 conn(0x7f02d806b4e0 0x7f02d806b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d80cc030 con 0x7f02d813d080 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f02c8001670 con 0x7f02d813d080 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 >> v1:172.21.15.124:6789/0 conn(0x7f02d813d080 legacy=0x7f02d815aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02c0003970 0x7f02c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02c00047c0 0x7f02c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.701 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d813d460 con 0x7f02c00047c0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d806beb0 con 0x7f02c0003970 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02c000ba30 con 0x7f02d80ce3d0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d77fe640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02c00047c0 0x7f02c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02d7fff640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02c0003970 0x7f02c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.698+0000 7f02dca12640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 -- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 msgr2=0x7f02c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 -- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02c0003970 msgr2=0x7f02c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02c0003970 0x7f02c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 -- 172.21.15.124:0/1623629677 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02c0003520 con 0x7f02c00047c0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02dca12640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02d80ce3d0 0x7f02c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d77fe640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02c00047c0 0x7f02c0004c40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00ba00 tx=0x7f02c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b04a8a799a6af5a4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032630 con 0x7f02c00047c0 2023-12-13T23:57:18.702 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc032790 con 0x7f02c00047c0 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.699+0000 7f02d6ffd640 1 -- 172.21.15.124:0/1623629677 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc032b20 con 0x7f02c00047c0 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 -- 172.21.15.124:0/1623629677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02c00047c0 msgr2=0x7f02c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 --2- 172.21.15.124:0/1623629677 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02c00047c0 0x7f02c0004c40 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f02cc00ba00 tx=0x7f02c00057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02dca12640 1 -- 172.21.15.124:0/1623629677 reap_dead start 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 -- 172.21.15.124:0/1623629677 shutdown_connections 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 -- 172.21.15.124:0/1623629677 >> 172.21.15.124:0/1623629677 conn(0x7f02d8070e70 msgr2=0x7f02d8071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 -- 172.21.15.124:0/1623629677 shutdown_connections 2023-12-13T23:57:18.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.700+0000 7f02de475640 1 -- 172.21.15.124:0/1623629677 wait complete. 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.701+0000 7f02de475640 1 Processor -- start 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.701+0000 7f02de475640 1 -- start start 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02c00047c0 0x7f02d8143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02d813b0f0 0x7f02d813b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d815ad40 con 0x7f02d8138b60 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d813bab0 con 0x7f02d813b0f0 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02de475640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d8156560 con 0x7f02c00047c0 2023-12-13T23:57:18.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45792/0 (socket says 172.21.15.124:45792) 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 -- 172.21.15.124:0/435499991 learned_addr learned my addr 172.21.15.124:0/435499991 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 -- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02c00047c0 msgr2=0x7f02d8143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02c00047c0 0x7f02d8143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 -- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02d813b0f0 msgr2=0x7f02d813b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02d813b0f0 0x7f02d813b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.702+0000 7f02d77fe640 1 -- 172.21.15.124:0/435499991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d8156840 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02d77fe640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02cc02beb0 tx=0x7f02cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc004220 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02cc0043a0 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d815a680 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02cc03b9f0 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.703+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d8156b20 con 0x7f02d8138b60 2023-12-13T23:57:18.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.704+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02d8069400 con 0x7f02d8138b60 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.704+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f02cc03bb50 con 0x7f02d8138b60 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.704+0000 7f02d4ff9640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 0x7f02c406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.704+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02cc0ac1b0 con 0x7f02d8138b60 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.705+0000 7f02d7fff640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 0x7f02c406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.705+0000 7f02d7fff640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 0x7f02c406d6b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02c8001b70 tx=0x7f02c8001b00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:18.708 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.708+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f02cc076f70 con 0x7f02d8138b60 2023-12-13T23:57:18.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.911+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f02d8069400 con 0x7f02d8138b60 2023-12-13T23:57:18.997 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.997+0000 7f02d4ff9640 1 -- 172.21.15.124:0/435499991 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v21) v1 ==== 118+0+0 (secure 0 0 0) 0x7f02cc07a590 con 0x7f02d8138b60 2023-12-13T23:57:19.000 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 msgr2=0x7f02c406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.000 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 0x7f02c406d6b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f02c8001b70 tx=0x7f02c8001b00 comp rx=0 tx=0).stop 2023-12-13T23:57:19.000 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 msgr2=0x7f02d8138320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.000 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f02cc02beb0 tx=0x7f02cc004730 comp rx=0 tx=0).stop 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 shutdown_connections 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f02c406b1f0 0x7f02c406d6b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f02d813b0f0 0x7f02d813b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02d8138b60 0x7f02d8138320 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 --2- 172.21.15.124:0/435499991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f02c00047c0 0x7f02d8143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:18.999+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 >> 172.21.15.124:0/435499991 conn(0x7f02d8070e70 msgr2=0x7f02d8071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:19.002 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.000+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 shutdown_connections 2023-12-13T23:57:19.002 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.000+0000 7f02de475640 1 -- 172.21.15.124:0/435499991 wait complete. 2023-12-13T23:57:19.010 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-12-13T23:57:19.010 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-12-13T23:57:19.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.112+0000 7faffc716640 1 Processor -- start 2023-12-13T23:57:19.112 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.112+0000 7faffc716640 1 -- start start 2023-12-13T23:57:19.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.113+0000 7faffc716640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faff4143490 0x7faff40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.113+0000 7faffc716640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7faff415abf0 0x7faff40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.113+0000 7faffc716640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff415e2a0 con 0x7faff415e730 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.113+0000 7faffc716640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff415b520 con 0x7faff4143490 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.113+0000 7faffc716640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff4143870 con 0x7faff415abf0 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffa4b2640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faff4143490 0x7faff40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faff9cb1640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7faff415e730 0x7faff415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47660/0 (socket says 172.21.15.124:47660) 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faff9cb1640 1 -- 172.21.15.124:0/2154651340 learned_addr learned my addr 172.21.15.124:0/2154651340 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffacb3640 1 --2- 172.21.15.124:0/2154651340 >> v2:172.21.15.203:3300/0 conn(0x7faff415abf0 0x7faff40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffa4b2640 1 -- 172.21.15.124:0/2154651340 >> v2:172.21.15.203:3300/0 conn(0x7faff415abf0 msgr2=0x7faff40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffa4b2640 1 --2- 172.21.15.124:0/2154651340 >> v2:172.21.15.203:3300/0 conn(0x7faff415abf0 0x7faff40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffa4b2640 1 -- 172.21.15.124:0/2154651340 >> v1:172.21.15.124:6789/0 conn(0x7faff415e730 legacy=0x7faff415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.114+0000 7faffa4b2640 1 -- 172.21.15.124:0/2154651340 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff415b240 con 0x7faff4143490 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffa4b2640 1 --2- 172.21.15.124:0/2154651340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff4143490 0x7faff40b0cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faff40b4ca0 tx=0x7faff002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da7a7f5056926f5d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:19.116 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faff94b0640 1 -- 172.21.15.124:0/2154651340 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0031830 con 0x7faff4143490 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faff94b0640 1 -- 172.21.15.124:0/2154651340 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0031990 con 0x7faff4143490 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faff94b0640 1 -- 172.21.15.124:0/2154651340 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0031d20 con 0x7faff4143490 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 -- 172.21.15.124:0/2154651340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff4143490 msgr2=0x7faff40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 --2- 172.21.15.124:0/2154651340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff4143490 0x7faff40b0cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faff40b4ca0 tx=0x7faff002b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 -- 172.21.15.124:0/2154651340 shutdown_connections 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 --2- 172.21.15.124:0/2154651340 >> v2:172.21.15.203:3300/0 conn(0x7faff415abf0 0x7faff40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 --2- 172.21.15.124:0/2154651340 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff4143490 0x7faff40b0cf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 -- 172.21.15.124:0/2154651340 >> 172.21.15.124:0/2154651340 conn(0x7faff406a4d0 msgr2=0x7faff406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 -- 172.21.15.124:0/2154651340 shutdown_connections 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.115+0000 7faffc716640 1 -- 172.21.15.124:0/2154651340 wait complete. 2023-12-13T23:57:19.117 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 Processor -- start 2023-12-13T23:57:19.118 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 -- start start 2023-12-13T23:57:19.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faff4143490 0x7faff4159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 0x7faff40b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40a8b10 con 0x7faff415abf0 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff40a8df0 con 0x7faff4143490 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faffc716640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff40a90d0 con 0x7faff415e730 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faff9cb1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faff9cb1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45804/0 (socket says 172.21.15.124:45804) 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.117+0000 7faff9cb1640 1 -- 172.21.15.124:0/1480524257 learned_addr learned my addr 172.21.15.124:0/1480524257 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faffa4b2640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faff4143490 0x7faff4159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faffacb3640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 0x7faff40b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 -- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 msgr2=0x7faff40b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 0x7faff40b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 -- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faff4143490 msgr2=0x7faff4159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faff4143490 0x7faff4159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 -- 172.21.15.124:0/1480524257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0009400 con 0x7faff415abf0 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faffacb3640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 0x7faff40b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faff9cb1640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c480 tx=0x7fafe400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4005ce0 con 0x7faff415abf0 2023-12-13T23:57:19.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4005e40 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe40072f0 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff40a93b0 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.118+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff40a9970 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.119+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff40a9b90 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.119+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fafe4007510 con 0x7faff415abf0 2023-12-13T23:57:19.123 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.120+0000 7fafeaffd640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 0x7fafd806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.120+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fafe4088e10 con 0x7faff415abf0 2023-12-13T23:57:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.120+0000 7faffa4b2640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 0x7fafd806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:19.124 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.121+0000 7faffa4b2640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 0x7fafd806d680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff003c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:19.125 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.124+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fafe4053c80 con 0x7faff415abf0 2023-12-13T23:57:19.325 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.326+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7faff40b4f40 con 0x7faff415abf0 2023-12-13T23:57:19.959 INFO:tasks.ceph.mds.b.smithi203.stderr:2023-12-13T23:57:19.960+0000 7f61035c6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:57:19.998 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:19.999+0000 7fafeaffd640 1 -- 172.21.15.124:0/1480524257 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafe4017020 con 0x7faff415abf0 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 msgr2=0x7fafd806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 0x7fafd806d680 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7faff0036000 tx=0x7faff003c040 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 msgr2=0x7faff40b0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c480 tx=0x7fafe400c940 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 shutdown_connections 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd806b1c0 0x7fafd806d680 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7faff415e730 0x7faff40b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faff415abf0 0x7faff40b0060 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 --2- 172.21.15.124:0/1480524257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7faff4143490 0x7faff4159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 >> 172.21.15.124:0/1480524257 conn(0x7faff406a4d0 msgr2=0x7faff4071980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.001+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 shutdown_connections 2023-12-13T23:57:20.001 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.002+0000 7faffc716640 1 -- 172.21.15.124:0/1480524257 wait complete. 2023-12-13T23:57:20.013 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-12-13T23:57:20.013 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:20.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.115+0000 7f20f4397640 1 Processor -- start 2023-12-13T23:57:20.115 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.115+0000 7f20f4397640 1 -- start start 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f4397640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f20ec15e730 0x7f20ec15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f4397640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec15e2a0 con 0x7f20ec143490 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f4397640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20ec15b240 con 0x7f20ec0b1400 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f4397640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20ec15b520 con 0x7f20ec15e730 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f2934640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f20ec15e730 0x7f20ec15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.119 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f2133640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f20ec143490 0x7f20ec0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47666/0 (socket says 172.21.15.124:47666) 2023-12-13T23:57:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.116+0000 7f20f2133640 1 -- 172.21.15.124:0/859038283 learned_addr learned my addr 172.21.15.124:0/859038283 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.117+0000 7f20f2934640 1 -- 172.21.15.124:0/859038283 >> v1:172.21.15.203:6789/0 conn(0x7f20ec0b1400 legacy=0x7f20ec15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.117+0000 7f20f2934640 1 -- 172.21.15.124:0/859038283 >> v1:172.21.15.124:6789/0 conn(0x7f20ec143490 legacy=0x7f20ec0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.120 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.117+0000 7f20f1131640 1 -- 172.21.15.124:0/859038283 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511316266 0 0) 0x7f20ec15e2a0 con 0x7f20ec143490 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.117+0000 7f20f2934640 1 -- 172.21.15.124:0/859038283 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20ec15dc50 con 0x7f20ec15e730 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.117+0000 7f20f2934640 1 --2- 172.21.15.124:0/859038283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec15e730 0x7f20ec15abf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f20e400bfa0 tx=0x7f20e402e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a09ae70db3c658 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f1131640 1 -- 172.21.15.124:0/859038283 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e402eae0 con 0x7f20ec15e730 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f1131640 1 -- 172.21.15.124:0/859038283 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e402ec40 con 0x7f20ec15e730 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f1131640 1 -- 172.21.15.124:0/859038283 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e4009540 con 0x7f20ec15e730 2023-12-13T23:57:20.121 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 -- 172.21.15.124:0/859038283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec15e730 msgr2=0x7f20ec15abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 --2- 172.21.15.124:0/859038283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec15e730 0x7f20ec15abf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f20e400bfa0 tx=0x7f20e402e140 comp rx=0 tx=0).stop 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 -- 172.21.15.124:0/859038283 shutdown_connections 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 --2- 172.21.15.124:0/859038283 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec15e730 0x7f20ec15abf0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 -- 172.21.15.124:0/859038283 >> 172.21.15.124:0/859038283 conn(0x7f20ec06a4d0 msgr2=0x7f20ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 -- 172.21.15.124:0/859038283 shutdown_connections 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.118+0000 7f20f4397640 1 -- 172.21.15.124:0/859038283 wait complete. 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.119+0000 7f20f4397640 1 Processor -- start 2023-12-13T23:57:20.122 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.119+0000 7f20f4397640 1 -- start start 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 0x7f20ec13bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20ec15e730 0x7f20ec0af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec159d90 con 0x7f20ec0b1400 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec13acb0 con 0x7f20ec143490 2023-12-13T23:57:20.126 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f4397640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20ec13af90 con 0x7f20ec15e730 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f2133640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f2133640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45822/0 (socket says 172.21.15.124:45822) 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.120+0000 7f20f2133640 1 -- 172.21.15.124:0/1677644180 learned_addr learned my addr 172.21.15.124:0/1677644180 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2934640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20ec15e730 0x7f20ec0af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f1932640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 0x7f20ec13bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 -- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20ec15e730 msgr2=0x7f20ec0af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20ec15e730 0x7f20ec0af5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 -- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 msgr2=0x7f20ec13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.127 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 0x7f20ec13bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 -- 172.21.15.124:0/1677644180 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e400bc80 con 0x7f20ec0b1400 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f1932640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 0x7f20ec13bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f2133640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c4a0 tx=0x7f20e800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e80074e0 con 0x7f20ec0b1400 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20e8007640 con 0x7f20ec0b1400 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20ec13b270 con 0x7f20ec0b1400 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20e8004a00 con 0x7f20ec0b1400 2023-12-13T23:57:20.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.121+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20ec13b830 con 0x7f20ec0b1400 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.122+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20ec069400 con 0x7f20ec0b1400 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.122+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f20e8004b60 con 0x7f20ec0b1400 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.122+0000 7f20e2ffd640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 0x7f20d006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.123+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f20e8089030 con 0x7f20ec0b1400 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.123+0000 7f20f1932640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 0x7f20d006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.123+0000 7f20f1932640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 0x7f20d006d680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f20ec1599d0 tx=0x7f20dc009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.129 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.127+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f20e8053cd0 con 0x7f20ec0b1400 2023-12-13T23:57:20.326 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.327+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20ec0b2820 con 0x7f20ec0b1400 2023-12-13T23:57:20.327 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.328+0000 7f20e2ffd640 1 -- 172.21.15.124:0/1677644180 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f20e8018020 con 0x7f20ec0b1400 2023-12-13T23:57:20.327 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:20.327 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:20.328 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:20.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 msgr2=0x7f20d006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 0x7f20d006d680 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f20ec1599d0 tx=0x7f20dc009040 comp rx=0 tx=0).stop 2023-12-13T23:57:20.329 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 msgr2=0x7f20ec13a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f20e800c4a0 tx=0x7f20e800c960 comp rx=0 tx=0).stop 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 shutdown_connections 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f20d006b1c0 0x7f20d006d680 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f20ec143490 0x7f20ec13bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f20ec15e730 0x7f20ec0af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 --2- 172.21.15.124:0/1677644180 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f20ec0b1400 0x7f20ec13a6d0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 >> 172.21.15.124:0/1677644180 conn(0x7f20ec06a4d0 msgr2=0x7f20ec071980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 shutdown_connections 2023-12-13T23:57:20.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.330+0000 7f20f4397640 1 -- 172.21.15.124:0/1677644180 wait complete. 2023-12-13T23:57:20.340 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:20.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.440+0000 7f6dc5cbb640 1 Processor -- start 2023-12-13T23:57:20.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 -- start start 2023-12-13T23:57:20.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6dc00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6dc015abf0 0x7f6dc00b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc015e2a0 con 0x7f6dc015e730 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc015b520 con 0x7f6dc015abf0 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.441+0000 7f6dc5cbb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0143870 con 0x7f6dc0143490 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbeffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f6dc015e730 0x7f6dc015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:47680/0 (socket says 172.21.15.124:47680) 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbeffd640 1 -- 172.21.15.124:0/207676436 learned_addr learned my addr 172.21.15.124:0/207676436 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6dc00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbffff640 1 --2- 172.21.15.124:0/207676436 >> v2:172.21.15.203:3300/0 conn(0x7f6dc015abf0 0x7f6dc00b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539343511 0 0) 0x7f6dc015e2a0 con 0x7f6dc015e730 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da8003520 con 0x7f6dc015e730 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2933051835 0 0) 0x7f6da8003520 con 0x7f6dc015e730 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 msgr2=0x7f6dc00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6dc00b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 >> v2:172.21.15.203:3300/0 conn(0x7f6dc015abf0 msgr2=0x7f6dc00b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 --2- 172.21.15.124:0/207676436 >> v2:172.21.15.203:3300/0 conn(0x7f6dc015abf0 0x7f6dc00b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.444 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc015b240 con 0x7f6dc015e730 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f6db4001670 con 0x7f6dc015e730 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 >> v1:172.21.15.124:6789/0 conn(0x7f6dc015e730 legacy=0x7f6dc015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6da8003970 0x7f6da8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6da80047c0 0x7f6da8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6da8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc015b520 con 0x7f6da8003970 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0143870 con 0x7f6da80047c0 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da800b980 con 0x7f6dc0143490 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6da8003970 0x7f6da8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbeffd640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6da80047c0 0x7f6da8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.442+0000 7f6dbffff640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6da8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 -- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 msgr2=0x7f6da8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6da8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 -- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6da80047c0 msgr2=0x7f6da8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6da80047c0 0x7f6da8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 -- 172.21.15.124:0/207676436 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da8003520 con 0x7f6da8003970 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbffff640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6dc0143490 0x7f6da8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbeffd640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6da80047c0 0x7f6da8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6da8003970 0x7f6da8003d90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009ac0 tx=0x7f6da80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17d45c248ec2982 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db002c6b0 con 0x7f6da8003970 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db002c810 con 0x7f6da8003970 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.443+0000 7f6dbe7fc640 1 -- 172.21.15.124:0/207676436 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db002cba0 con 0x7f6da8003970 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/207676436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6da8003970 msgr2=0x7f6da8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/207676436 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6da8003970 0x7f6da8003d90 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009ac0 tx=0x7f6da80057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dbffff640 1 -- 172.21.15.124:0/207676436 reap_dead start 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/207676436 shutdown_connections 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/207676436 >> 172.21.15.124:0/207676436 conn(0x7f6dc006a4d0 msgr2=0x7f6dc006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/207676436 shutdown_connections 2023-12-13T23:57:20.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.444+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/207676436 wait complete. 2023-12-13T23:57:20.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.445+0000 7f6dc5cbb640 1 Processor -- start 2023-12-13T23:57:20.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.445+0000 7f6dc5cbb640 1 -- start start 2023-12-13T23:57:20.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 0x7f6dc0159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 0x7f6dc013e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc013de30 con 0x7f6dc01e5540 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc013d220 con 0x7f6dc013e2c0 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dc5cbb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc013d500 con 0x7f6da8003970 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:45844/0 (socket says 172.21.15.124:45844) 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 -- 172.21.15.124:0/1723278887 learned_addr learned my addr 172.21.15.124:0/1723278887 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 0x7f6dc0159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbffff640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 0x7f6dc013e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 -- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 msgr2=0x7f6dc0159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 0x7f6dc0159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 -- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 msgr2=0x7f6dc013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 0x7f6dc013e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbeffd640 1 -- 172.21.15.124:0/1723278887 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc013d7e0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbffff640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 0x7f6dc013e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.446+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 0x7f6dc0159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6dbeffd640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6db400e7b0 tx=0x7f6db400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db400c8d0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db400ca30 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db400cdc0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc013dac0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.447+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc013cdf0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.448+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc013dfe0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.448+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6db401d460 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.448+0000 7f6da7fff640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 0x7f6dac06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.449+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6db40899e0 con 0x7f6dc01e5540 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.449+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 0x7f6dac06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.449+0000 7f6dbf7fe640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 0x7f6dac06d6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6db0036000 tx=0x7f6db0030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.453+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6dc013dfe0 con 0x7f6dc01e5540 2023-12-13T23:57:20.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.620+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6dc0069400 con 0x7f6dc01e5540 2023-12-13T23:57:20.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.621+0000 7f6da7fff640 1 -- 172.21.15.124:0/1723278887 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f6db4054710 con 0x7f6dc01e5540 2023-12-13T23:57:20.620 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:20.620 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 msgr2=0x7f6dac06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 0x7f6dac06d6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6db0036000 tx=0x7f6db0030000 comp rx=0 tx=0).stop 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 msgr2=0x7f6dc013c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6db400e7b0 tx=0x7f6db400ec70 comp rx=0 tx=0).stop 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 shutdown_connections 2023-12-13T23:57:20.622 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6dac06b1f0 0x7f6dac06d6b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6dc013e2c0 0x7f6dc013e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6dc01e5540 0x7f6dc013c8b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 --2- 172.21.15.124:0/1723278887 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6da8003970 0x7f6dc0159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 >> 172.21.15.124:0/1723278887 conn(0x7f6dc006a4d0 msgr2=0x7f6dc00aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 shutdown_connections 2023-12-13T23:57:20.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.623+0000 7f6dc5cbb640 1 -- 172.21.15.124:0/1723278887 wait complete. 2023-12-13T23:57:20.633 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-12-13T23:57:20.633 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:20.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.733+0000 7f492d668640 1 Processor -- start 2023-12-13T23:57:20.733 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.733+0000 7f492d668640 1 -- start start 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f492d668640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49280b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f492d668640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280b2b10 con 0x7f49280d2910 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f492d668640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49280d1590 con 0x7f492815d330 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f492d668640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49280d1870 con 0x7f49280d3900 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f49267fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f49280d2910 0x7f49280d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59186/0 (socket says 172.21.15.124:59186) 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f49267fc640 1 -- 172.21.15.124:0/1663860258 learned_addr learned my addr 172.21.15.124:0/1663860258 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693411858 0 0) 0x7f49280b2b10 con 0x7f49280d2910 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.734+0000 7f49277fe640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49280b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910003520 con 0x7f49280d2910 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216204826 0 0) 0x7f4910003520 con 0x7f49280d2910 2023-12-13T23:57:20.736 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 msgr2=0x7f49280b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49280b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 >> v1:172.21.15.203:6789/0 conn(0x7f492815d330 legacy=0x7f49280b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280d1b50 con 0x7f49280d2910 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f491c001670 con 0x7f49280d2910 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 >> v1:172.21.15.124:6789/0 conn(0x7f49280d2910 legacy=0x7f49280d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49100039a0 0x7f4910003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49100047c0 0x7f4910004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49100051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280d1870 con 0x7f49100039a0 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491000bab0 con 0x7f49100047c0 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491000bc70 con 0x7f49280d3900 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49100039a0 0x7f4910003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f49267fc640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49100047c0 0x7f4910004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.737 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f49277fe640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49100051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 -- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 msgr2=0x7f49100051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49280d3900 0x7f49100051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 -- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49100047c0 msgr2=0x7f4910004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:20.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49100047c0 0x7f4910004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.738 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f4926ffd640 1 -- 172.21.15.124:0/1663860258 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4910003520 con 0x7f49100039a0 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.735+0000 7f49267fc640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f49100047c0 0x7f4910004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.736+0000 7f4926ffd640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49100039a0 0x7f4910003dc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4918001750 tx=0x7f49100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b1f0e935b7d74c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.736+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180052d0 con 0x7f49100039a0 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.736+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f491802e050 con 0x7f49100039a0 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.736+0000 7f4925ffb640 1 -- 172.21.15.124:0/1663860258 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49180035f0 con 0x7f49100039a0 2023-12-13T23:57:20.739 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 -- 172.21.15.124:0/1663860258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49100039a0 msgr2=0x7f4910003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 --2- 172.21.15.124:0/1663860258 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49100039a0 0x7f4910003dc0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f4918001750 tx=0x7f49100057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f49277fe640 1 -- 172.21.15.124:0/1663860258 reap_dead start 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 -- 172.21.15.124:0/1663860258 shutdown_connections 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 -- 172.21.15.124:0/1663860258 >> 172.21.15.124:0/1663860258 conn(0x7f4928072020 msgr2=0x7f4928072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 -- 172.21.15.124:0/1663860258 shutdown_connections 2023-12-13T23:57:20.740 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.737+0000 7f492d668640 1 -- 172.21.15.124:0/1663860258 wait complete. 2023-12-13T23:57:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.738+0000 7f492d668640 1 Processor -- start 2023-12-13T23:57:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.738+0000 7f492d668640 1 -- start start 2023-12-13T23:57:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49100039a0 0x7f49280c5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4928155630 0x7f4928157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.743 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49280c6370 con 0x7f4928157ca0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4928155b80 con 0x7f4928155630 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4928155e60 con 0x7f49100039a0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57080/0 (socket says 172.21.15.124:57080) 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 -- 172.21.15.124:0/3817255032 learned_addr learned my addr 172.21.15.124:0/3817255032 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f4926ffd640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49100039a0 0x7f49280c5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49267fc640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4928155630 0x7f4928157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 -- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49100039a0 msgr2=0x7f49280c5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49100039a0 0x7f49280c5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 -- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4928155630 msgr2=0x7f4928157760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4928155630 0x7f4928157760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 -- 172.21.15.124:0/3817255032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928156140 con 0x7f4928157ca0 2023-12-13T23:57:20.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f49277fe640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f49280d1510 tx=0x7f49200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f492000e070 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4920011ed0 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4928156420 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4920003990 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.739+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4928159a30 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.740+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928069400 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.741+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4920003b10 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.741+0000 7f490f7fe640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 0x7f491406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.741+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4920089f80 con 0x7f4928157ca0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.741+0000 7f4926ffd640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 0x7f491406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:20.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.742+0000 7f4926ffd640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 0x7f491406d6b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4918033000 tx=0x7f4918009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:20.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.745+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f49200bce90 con 0x7f4928157ca0 2023-12-13T23:57:20.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.910+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49280d23f0 con 0x7f491406b1f0 2023-12-13T23:57:20.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.918+0000 7f490f7fe640 1 -- 172.21.15.124:0/3817255032 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289603 (secure 0 0 0) 0x7f4918005350 con 0x7f491406b1f0 2023-12-13T23:57:20.921 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:20.922 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 msgr2=0x7f491406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 0x7f491406d6b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4918033000 tx=0x7f4918009030 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 msgr2=0x7f4928159310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f49280d1510 tx=0x7f49200115a0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 shutdown_connections 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f491406b1f0 0x7f491406d6b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4928155630 0x7f4928157760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4928157ca0 0x7f4928159310 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 --2- 172.21.15.124:0/3817255032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f49100039a0 0x7f49280c5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 >> 172.21.15.124:0/3817255032 conn(0x7f4928072020 msgr2=0x7f49280b4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 shutdown_connections 2023-12-13T23:57:20.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:20.921+0000 7f492d668640 1 -- 172.21.15.124:0/3817255032 wait complete. 2023-12-13T23:57:20.931 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2023-12-13T23:57:20.140742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222036,"kb_used_data":5840,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752684,"statfs":{"total":773094113280,"available":772866748416,"internally_reserved":0,"allocated":5980160,"data_stored":3145580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.162839"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861819+0000","last_change":"2023-12-13T23:57:17.769091+0000","last_active":"2023-12-13T23:57:19.861819+0000","last_peered":"2023-12-13T23:57:19.861819+0000","last_clean":"2023-12-13T23:57:19.861819+0000","last_became_active":"2023-12-13T23:57:17.768834+0000","last_became_peered":"2023-12-13T23:57:17.768834+0000","last_unstale":"2023-12-13T23:57:19.861819+0000","last_undegraded":"2023-12-13T23:57:19.861819+0000","last_fullsized":"2023-12-13T23:57:19.861819+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762196+0000","last_change":"2023-12-13T23:57:17.767922+0000","last_active":"2023-12-13T23:57:18.762196+0000","last_peered":"2023-12-13T23:57:18.762196+0000","last_clean":"2023-12-13T23:57:18.762196+0000","last_became_active":"2023-12-13T23:57:17.767220+0000","last_became_peered":"2023-12-13T23:57:17.767220+0000","last_unstale":"2023-12-13T23:57:18.762196+0000","last_undegraded":"2023-12-13T23:57:18.762196+0000","last_fullsized":"2023-12-13T23:57:18.762196+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337273+0000","last_change":"2023-12-13T23:57:17.771889+0000","last_active":"2023-12-13T23:57:19.337273+0000","last_peered":"2023-12-13T23:57:19.337273+0000","last_clean":"2023-12-13T23:57:19.337273+0000","last_became_active":"2023-12-13T23:57:17.771650+0000","last_became_peered":"2023-12-13T23:57:17.771650+0000","last_unstale":"2023-12-13T23:57:19.337273+0000","last_undegraded":"2023-12-13T23:57:19.337273+0000","last_fullsized":"2023-12-13T23:57:19.337273+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:42.851791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762137+0000","last_change":"2023-12-13T23:57:17.773173+0000","last_active":"2023-12-13T23:57:18.762137+0000","last_peered":"2023-12-13T23:57:18.762137+0000","last_clean":"2023-12-13T23:57:18.762137+0000","last_became_active":"2023-12-13T23:57:17.772654+0000","last_became_peered":"2023-12-13T23:57:17.772654+0000","last_unstale":"2023-12-13T23:57:18.762137+0000","last_undegraded":"2023-12-13T23:57:18.762137+0000","last_fullsized":"2023-12-13T23:57:18.762137+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761242+0000","last_change":"2023-12-13T23:57:17.761851+0000","last_active":"2023-12-13T23:57:18.761242+0000","last_peered":"2023-12-13T23:57:18.761242+0000","last_clean":"2023-12-13T23:57:18.761242+0000","last_became_active":"2023-12-13T23:57:17.761449+0000","last_became_peered":"2023-12-13T23:57:17.761449+0000","last_unstale":"2023-12-13T23:57:18.761242+0000","last_undegraded":"2023-12-13T23:57:18.761242+0000","last_fullsized":"2023-12-13T23:57:18.761242+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:06.342217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861540+0000","last_change":"2023-12-13T23:57:17.772648+0000","last_active":"2023-12-13T23:57:19.861540+0000","last_peered":"2023-12-13T23:57:19.861540+0000","last_clean":"2023-12-13T23:57:19.861540+0000","last_became_active":"2023-12-13T23:57:17.772311+0000","last_became_peered":"2023-12-13T23:57:17.772311+0000","last_unstale":"2023-12-13T23:57:19.861540+0000","last_undegraded":"2023-12-13T23:57:19.861540+0000","last_fullsized":"2023-12-13T23:57:19.861540+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337050+0000","last_change":"2023-12-13T23:57:17.770329+0000","last_active":"2023-12-13T23:57:19.337050+0000","last_peered":"2023-12-13T23:57:19.337050+0000","last_clean":"2023-12-13T23:57:19.337050+0000","last_became_active":"2023-12-13T23:57:17.767466+0000","last_became_peered":"2023-12-13T23:57:17.767466+0000","last_unstale":"2023-12-13T23:57:19.337050+0000","last_undegraded":"2023-12-13T23:57:19.337050+0000","last_fullsized":"2023-12-13T23:57:19.337050+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861583+0000","last_change":"2023-12-13T23:57:17.780221+0000","last_active":"2023-12-13T23:57:19.861583+0000","last_peered":"2023-12-13T23:57:19.861583+0000","last_clean":"2023-12-13T23:57:19.861583+0000","last_became_active":"2023-12-13T23:57:17.779947+0000","last_became_peered":"2023-12-13T23:57:17.779947+0000","last_unstale":"2023-12-13T23:57:19.861583+0000","last_undegraded":"2023-12-13T23:57:19.861583+0000","last_fullsized":"2023-12-13T23:57:19.861583+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:20:25.103095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862215+0000","last_change":"2023-12-13T23:57:17.780203+0000","last_active":"2023-12-13T23:57:19.862215+0000","last_peered":"2023-12-13T23:57:19.862215+0000","last_clean":"2023-12-13T23:57:19.862215+0000","last_became_active":"2023-12-13T23:57:17.779908+0000","last_became_peered":"2023-12-13T23:57:17.779908+0000","last_unstale":"2023-12-13T23:57:19.862215+0000","last_undegraded":"2023-12-13T23:57:19.862215+0000","last_fullsized":"2023-12-13T23:57:19.862215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:11:47.949638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861818+0000","last_change":"2023-12-13T23:57:17.769718+0000","last_active":"2023-12-13T23:57:19.861818+0000","last_peered":"2023-12-13T23:57:19.861818+0000","last_clean":"2023-12-13T23:57:19.861818+0000","last_became_active":"2023-12-13T23:57:17.769436+0000","last_became_peered":"2023-12-13T23:57:17.769436+0000","last_unstale":"2023-12-13T23:57:19.861818+0000","last_undegraded":"2023-12-13T23:57:19.861818+0000","last_fullsized":"2023-12-13T23:57:19.861818+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760066+0000","last_change":"2023-12-13T23:57:17.771798+0000","last_active":"2023-12-13T23:57:18.760066+0000","last_peered":"2023-12-13T23:57:18.760066+0000","last_clean":"2023-12-13T23:57:18.760066+0000","last_became_active":"2023-12-13T23:57:17.771543+0000","last_became_peered":"2023-12-13T23:57:17.771543+0000","last_unstale":"2023-12-13T23:57:18.760066+0000","last_undegraded":"2023-12-13T23:57:18.760066+0000","last_fullsized":"2023-12-13T23:57:18.760066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:40:05.961525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862100+0000","last_change":"2023-12-13T23:57:17.769855+0000","last_active":"2023-12-13T23:57:19.862100+0000","last_peered":"2023-12-13T23:57:19.862100+0000","last_clean":"2023-12-13T23:57:19.862100+0000","last_became_active":"2023-12-13T23:57:17.769499+0000","last_became_peered":"2023-12-13T23:57:17.769499+0000","last_unstale":"2023-12-13T23:57:19.862100+0000","last_undegraded":"2023-12-13T23:57:19.862100+0000","last_fullsized":"2023-12-13T23:57:19.862100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:37:36.347678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861473+0000","last_change":"2023-12-13T23:57:17.769967+0000","last_active":"2023-12-13T23:57:19.861473+0000","last_peered":"2023-12-13T23:57:19.861473+0000","last_clean":"2023-12-13T23:57:19.861473+0000","last_became_active":"2023-12-13T23:57:17.769443+0000","last_became_peered":"2023-12-13T23:57:17.769443+0000","last_unstale":"2023-12-13T23:57:19.861473+0000","last_undegraded":"2023-12-13T23:57:19.861473+0000","last_fullsized":"2023-12-13T23:57:19.861473+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:55:00.239860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337219+0000","last_change":"2023-12-13T23:57:17.774073+0000","last_active":"2023-12-13T23:57:19.337219+0000","last_peered":"2023-12-13T23:57:19.337219+0000","last_clean":"2023-12-13T23:57:19.337219+0000","last_became_active":"2023-12-13T23:57:17.768211+0000","last_became_peered":"2023-12-13T23:57:17.768211+0000","last_unstale":"2023-12-13T23:57:19.337219+0000","last_undegraded":"2023-12-13T23:57:19.337219+0000","last_fullsized":"2023-12-13T23:57:19.337219+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761384+0000","last_change":"2023-12-13T23:57:17.770058+0000","last_active":"2023-12-13T23:57:18.761384+0000","last_peered":"2023-12-13T23:57:18.761384+0000","last_clean":"2023-12-13T23:57:18.761384+0000","last_became_active":"2023-12-13T23:57:17.769678+0000","last_became_peered":"2023-12-13T23:57:17.769678+0000","last_unstale":"2023-12-13T23:57:18.761384+0000","last_undegraded":"2023-12-13T23:57:18.761384+0000","last_fullsized":"2023-12-13T23:57:18.761384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861379+0000","last_change":"2023-12-13T23:57:17.771706+0000","last_active":"2023-12-13T23:57:19.861379+0000","last_peered":"2023-12-13T23:57:19.861379+0000","last_clean":"2023-12-13T23:57:19.861379+0000","last_became_active":"2023-12-13T23:57:17.771466+0000","last_became_peered":"2023-12-13T23:57:17.771466+0000","last_unstale":"2023-12-13T23:57:19.861379+0000","last_undegraded":"2023-12-13T23:57:19.861379+0000","last_fullsized":"2023-12-13T23:57:19.861379+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:42.626381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337534+0000","last_change":"2023-12-13T23:57:17.772457+0000","last_active":"2023-12-13T23:57:19.337534+0000","last_peered":"2023-12-13T23:57:19.337534+0000","last_clean":"2023-12-13T23:57:19.337534+0000","last_became_active":"2023-12-13T23:57:17.772211+0000","last_became_peered":"2023-12-13T23:57:17.772211+0000","last_unstale":"2023-12-13T23:57:19.337534+0000","last_undegraded":"2023-12-13T23:57:19.337534+0000","last_fullsized":"2023-12-13T23:57:19.337534+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:07:01.784038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762266+0000","last_change":"2023-12-13T23:57:17.771231+0000","last_active":"2023-12-13T23:57:18.762266+0000","last_peered":"2023-12-13T23:57:18.762266+0000","last_clean":"2023-12-13T23:57:18.762266+0000","last_became_active":"2023-12-13T23:57:17.770956+0000","last_became_peered":"2023-12-13T23:57:17.770956+0000","last_unstale":"2023-12-13T23:57:18.762266+0000","last_undegraded":"2023-12-13T23:57:18.762266+0000","last_fullsized":"2023-12-13T23:57:18.762266+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:14.954573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337335+0000","last_change":"2023-12-13T23:57:17.775732+0000","last_active":"2023-12-13T23:57:19.337335+0000","last_peered":"2023-12-13T23:57:19.337335+0000","last_clean":"2023-12-13T23:57:19.337335+0000","last_became_active":"2023-12-13T23:57:17.775357+0000","last_became_peered":"2023-12-13T23:57:17.775357+0000","last_unstale":"2023-12-13T23:57:19.337335+0000","last_undegraded":"2023-12-13T23:57:19.337335+0000","last_fullsized":"2023-12-13T23:57:19.337335+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:41:33.439472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530542+0000","last_change":"2023-12-13T23:57:17.770778+0000","last_active":"2023-12-13T23:57:19.530542+0000","last_peered":"2023-12-13T23:57:19.530542+0000","last_clean":"2023-12-13T23:57:19.530542+0000","last_became_active":"2023-12-13T23:57:17.770323+0000","last_became_peered":"2023-12-13T23:57:17.770323+0000","last_unstale":"2023-12-13T23:57:19.530542+0000","last_undegraded":"2023-12-13T23:57:19.530542+0000","last_fullsized":"2023-12-13T23:57:19.530542+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:46.163512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530460+0000","last_change":"2023-12-13T23:57:17.770750+0000","last_active":"2023-12-13T23:57:19.530460+0000","last_peered":"2023-12-13T23:57:19.530460+0000","last_clean":"2023-12-13T23:57:19.530460+0000","last_became_active":"2023-12-13T23:57:17.770470+0000","last_became_peered":"2023-12-13T23:57:17.770470+0000","last_unstale":"2023-12-13T23:57:19.530460+0000","last_undegraded":"2023-12-13T23:57:19.530460+0000","last_fullsized":"2023-12-13T23:57:19.530460+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:24:51.585501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861444+0000","last_change":"2023-12-13T23:57:17.780310+0000","last_active":"2023-12-13T23:57:19.861444+0000","last_peered":"2023-12-13T23:57:19.861444+0000","last_clean":"2023-12-13T23:57:19.861444+0000","last_became_active":"2023-12-13T23:57:17.780111+0000","last_became_peered":"2023-12-13T23:57:17.780111+0000","last_unstale":"2023-12-13T23:57:19.861444+0000","last_undegraded":"2023-12-13T23:57:19.861444+0000","last_fullsized":"2023-12-13T23:57:19.861444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:17:24.759895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.759876+0000","last_change":"2023-12-13T23:57:17.771057+0000","last_active":"2023-12-13T23:57:18.759876+0000","last_peered":"2023-12-13T23:57:18.759876+0000","last_clean":"2023-12-13T23:57:18.759876+0000","last_became_active":"2023-12-13T23:57:17.770813+0000","last_became_peered":"2023-12-13T23:57:17.770813+0000","last_unstale":"2023-12-13T23:57:18.759876+0000","last_undegraded":"2023-12-13T23:57:18.759876+0000","last_fullsized":"2023-12-13T23:57:18.759876+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:05:52.770820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337013+0000","last_change":"2023-12-13T23:57:17.767485+0000","last_active":"2023-12-13T23:57:19.337013+0000","last_peered":"2023-12-13T23:57:19.337013+0000","last_clean":"2023-12-13T23:57:19.337013+0000","last_became_active":"2023-12-13T23:57:17.767236+0000","last_became_peered":"2023-12-13T23:57:17.767236+0000","last_unstale":"2023-12-13T23:57:19.337013+0000","last_undegraded":"2023-12-13T23:57:19.337013+0000","last_fullsized":"2023-12-13T23:57:19.337013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337535+0000","last_change":"2023-12-13T23:57:17.771759+0000","last_active":"2023-12-13T23:57:19.337535+0000","last_peered":"2023-12-13T23:57:19.337535+0000","last_clean":"2023-12-13T23:57:19.337535+0000","last_became_active":"2023-12-13T23:57:17.771490+0000","last_became_peered":"2023-12-13T23:57:17.771490+0000","last_unstale":"2023-12-13T23:57:19.337535+0000","last_undegraded":"2023-12-13T23:57:19.337535+0000","last_fullsized":"2023-12-13T23:57:19.337535+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:21:35.395021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861939+0000","last_change":"2023-12-13T23:57:17.772866+0000","last_active":"2023-12-13T23:57:19.861939+0000","last_peered":"2023-12-13T23:57:19.861939+0000","last_clean":"2023-12-13T23:57:19.861939+0000","last_became_active":"2023-12-13T23:57:17.772350+0000","last_became_peered":"2023-12-13T23:57:17.772350+0000","last_unstale":"2023-12-13T23:57:19.861939+0000","last_undegraded":"2023-12-13T23:57:19.861939+0000","last_fullsized":"2023-12-13T23:57:19.861939+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:54:07.211775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862042+0000","last_change":"2023-12-13T23:57:17.777353+0000","last_active":"2023-12-13T23:57:19.862042+0000","last_peered":"2023-12-13T23:57:19.862042+0000","last_clean":"2023-12-13T23:57:19.862042+0000","last_became_active":"2023-12-13T23:57:17.777104+0000","last_became_peered":"2023-12-13T23:57:17.777104+0000","last_unstale":"2023-12-13T23:57:19.862042+0000","last_undegraded":"2023-12-13T23:57:19.862042+0000","last_fullsized":"2023-12-13T23:57:19.862042+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530616+0000","last_change":"2023-12-13T23:57:17.771415+0000","last_active":"2023-12-13T23:57:19.530616+0000","last_peered":"2023-12-13T23:57:19.530616+0000","last_clean":"2023-12-13T23:57:19.530616+0000","last_became_active":"2023-12-13T23:57:17.771091+0000","last_became_peered":"2023-12-13T23:57:17.771091+0000","last_unstale":"2023-12-13T23:57:19.530616+0000","last_undegraded":"2023-12-13T23:57:19.530616+0000","last_fullsized":"2023-12-13T23:57:19.530616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:10:40.296399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337330+0000","last_change":"2023-12-13T23:57:17.768872+0000","last_active":"2023-12-13T23:57:19.337330+0000","last_peered":"2023-12-13T23:57:19.337330+0000","last_clean":"2023-12-13T23:57:19.337330+0000","last_became_active":"2023-12-13T23:57:17.768630+0000","last_became_peered":"2023-12-13T23:57:17.768630+0000","last_unstale":"2023-12-13T23:57:19.337330+0000","last_undegraded":"2023-12-13T23:57:19.337330+0000","last_fullsized":"2023-12-13T23:57:19.337330+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:16:08.664883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761749+0000","last_change":"2023-12-13T23:57:17.769015+0000","last_active":"2023-12-13T23:57:18.761749+0000","last_peered":"2023-12-13T23:57:18.761749+0000","last_clean":"2023-12-13T23:57:18.761749+0000","last_became_active":"2023-12-13T23:57:17.768743+0000","last_became_peered":"2023-12-13T23:57:17.768743+0000","last_unstale":"2023-12-13T23:57:18.761749+0000","last_undegraded":"2023-12-13T23:57:18.761749+0000","last_fullsized":"2023-12-13T23:57:18.761749+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337118+0000","last_change":"2023-12-13T23:57:17.768464+0000","last_active":"2023-12-13T23:57:19.337118+0000","last_peered":"2023-12-13T23:57:19.337118+0000","last_clean":"2023-12-13T23:57:19.337118+0000","last_became_active":"2023-12-13T23:57:17.767146+0000","last_became_peered":"2023-12-13T23:57:17.767146+0000","last_unstale":"2023-12-13T23:57:19.337118+0000","last_undegraded":"2023-12-13T23:57:19.337118+0000","last_fullsized":"2023-12-13T23:57:19.337118+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141425+0000","last_change":"2023-12-13T23:57:17.774634+0000","last_active":"2023-12-13T23:57:19.141425+0000","last_peered":"2023-12-13T23:57:19.141425+0000","last_clean":"2023-12-13T23:57:19.141425+0000","last_became_active":"2023-12-13T23:57:17.774342+0000","last_became_peered":"2023-12-13T23:57:17.774342+0000","last_unstale":"2023-12-13T23:57:19.141425+0000","last_undegraded":"2023-12-13T23:57:19.141425+0000","last_fullsized":"2023-12-13T23:57:19.141425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:45.776626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761890+0000","last_change":"2023-12-13T23:57:17.771633+0000","last_active":"2023-12-13T23:57:18.761890+0000","last_peered":"2023-12-13T23:57:18.761890+0000","last_clean":"2023-12-13T23:57:18.761890+0000","last_became_active":"2023-12-13T23:57:17.771365+0000","last_became_peered":"2023-12-13T23:57:17.771365+0000","last_unstale":"2023-12-13T23:57:18.761890+0000","last_undegraded":"2023-12-13T23:57:18.761890+0000","last_fullsized":"2023-12-13T23:57:18.761890+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530730+0000","last_change":"2023-12-13T23:57:16.996579+0000","last_active":"2023-12-13T23:57:19.530730+0000","last_peered":"2023-12-13T23:57:19.530730+0000","last_clean":"2023-12-13T23:57:19.530730+0000","last_became_active":"2023-12-13T23:57:16.996282+0000","last_became_peered":"2023-12-13T23:57:16.996282+0000","last_unstale":"2023-12-13T23:57:19.530730+0000","last_undegraded":"2023-12-13T23:57:19.530730+0000","last_fullsized":"2023-12-13T23:57:19.530730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:22:10.185846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141544+0000","last_change":"2023-12-13T23:57:17.775233+0000","last_active":"2023-12-13T23:57:19.141544+0000","last_peered":"2023-12-13T23:57:19.141544+0000","last_clean":"2023-12-13T23:57:19.141544+0000","last_became_active":"2023-12-13T23:57:17.774947+0000","last_became_peered":"2023-12-13T23:57:17.774947+0000","last_unstale":"2023-12-13T23:57:19.141544+0000","last_undegraded":"2023-12-13T23:57:19.141544+0000","last_fullsized":"2023-12-13T23:57:19.141544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:32.100913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861883+0000","last_change":"2023-12-13T23:57:16.997782+0000","last_active":"2023-12-13T23:57:19.861883+0000","last_peered":"2023-12-13T23:57:19.861883+0000","last_clean":"2023-12-13T23:57:19.861883+0000","last_became_active":"2023-12-13T23:57:16.997203+0000","last_became_peered":"2023-12-13T23:57:16.997203+0000","last_unstale":"2023-12-13T23:57:19.861883+0000","last_undegraded":"2023-12-13T23:57:19.861883+0000","last_fullsized":"2023-12-13T23:57:19.861883+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:27:40.143895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530463+0000","last_change":"2023-12-13T23:57:17.770837+0000","last_active":"2023-12-13T23:57:19.530463+0000","last_peered":"2023-12-13T23:57:19.530463+0000","last_clean":"2023-12-13T23:57:19.530463+0000","last_became_active":"2023-12-13T23:57:17.770461+0000","last_became_peered":"2023-12-13T23:57:17.770461+0000","last_unstale":"2023-12-13T23:57:19.530463+0000","last_undegraded":"2023-12-13T23:57:19.530463+0000","last_fullsized":"2023-12-13T23:57:19.530463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:31:23.636612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861859+0000","last_change":"2023-12-13T23:57:16.991830+0000","last_active":"2023-12-13T23:57:19.861859+0000","last_peered":"2023-12-13T23:57:19.861859+0000","last_clean":"2023-12-13T23:57:19.861859+0000","last_became_active":"2023-12-13T23:57:16.991558+0000","last_became_peered":"2023-12-13T23:57:16.991558+0000","last_unstale":"2023-12-13T23:57:19.861859+0000","last_undegraded":"2023-12-13T23:57:19.861859+0000","last_fullsized":"2023-12-13T23:57:19.861859+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861654+0000","last_change":"2023-12-13T23:57:17.772416+0000","last_active":"2023-12-13T23:57:19.861654+0000","last_peered":"2023-12-13T23:57:19.861654+0000","last_clean":"2023-12-13T23:57:19.861654+0000","last_became_active":"2023-12-13T23:57:17.772173+0000","last_became_peered":"2023-12-13T23:57:17.772173+0000","last_unstale":"2023-12-13T23:57:19.861654+0000","last_undegraded":"2023-12-13T23:57:19.861654+0000","last_fullsized":"2023-12-13T23:57:19.861654+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:24:48.909848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141617+0000","last_change":"2023-12-13T23:57:17.001547+0000","last_active":"2023-12-13T23:57:19.141617+0000","last_peered":"2023-12-13T23:57:19.141617+0000","last_clean":"2023-12-13T23:57:19.141617+0000","last_became_active":"2023-12-13T23:57:17.001344+0000","last_became_peered":"2023-12-13T23:57:17.001344+0000","last_unstale":"2023-12-13T23:57:19.141617+0000","last_undegraded":"2023-12-13T23:57:19.141617+0000","last_fullsized":"2023-12-13T23:57:19.141617+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761257+0000","last_change":"2023-12-13T23:57:17.768218+0000","last_active":"2023-12-13T23:57:18.761257+0000","last_peered":"2023-12-13T23:57:18.761257+0000","last_clean":"2023-12-13T23:57:18.761257+0000","last_became_active":"2023-12-13T23:57:17.767959+0000","last_became_peered":"2023-12-13T23:57:17.767959+0000","last_unstale":"2023-12-13T23:57:18.761257+0000","last_undegraded":"2023-12-13T23:57:18.761257+0000","last_fullsized":"2023-12-13T23:57:18.761257+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:10:40.296399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337167+0000","last_change":"2023-12-13T23:57:17.002065+0000","last_active":"2023-12-13T23:57:19.337167+0000","last_peered":"2023-12-13T23:57:19.337167+0000","last_clean":"2023-12-13T23:57:19.337167+0000","last_became_active":"2023-12-13T23:57:17.001815+0000","last_became_peered":"2023-12-13T23:57:17.001815+0000","last_unstale":"2023-12-13T23:57:19.337167+0000","last_undegraded":"2023-12-13T23:57:19.337167+0000","last_fullsized":"2023-12-13T23:57:19.337167+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:05.340307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762203+0000","last_change":"2023-12-13T23:57:17.768443+0000","last_active":"2023-12-13T23:57:18.762203+0000","last_peered":"2023-12-13T23:57:18.762203+0000","last_clean":"2023-12-13T23:57:18.762203+0000","last_became_active":"2023-12-13T23:57:17.768139+0000","last_became_peered":"2023-12-13T23:57:17.768139+0000","last_unstale":"2023-12-13T23:57:18.762203+0000","last_undegraded":"2023-12-13T23:57:18.762203+0000","last_fullsized":"2023-12-13T23:57:18.762203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761177+0000","last_change":"2023-12-13T23:57:16.999010+0000","last_active":"2023-12-13T23:57:18.761177+0000","last_peered":"2023-12-13T23:57:18.761177+0000","last_clean":"2023-12-13T23:57:18.761177+0000","last_became_active":"2023-12-13T23:57:16.998688+0000","last_became_peered":"2023-12-13T23:57:16.998688+0000","last_unstale":"2023-12-13T23:57:18.761177+0000","last_undegraded":"2023-12-13T23:57:18.761177+0000","last_fullsized":"2023-12-13T23:57:18.761177+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:39:14.365772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862106+0000","last_change":"2023-12-13T23:57:17.772583+0000","last_active":"2023-12-13T23:57:19.862106+0000","last_peered":"2023-12-13T23:57:19.862106+0000","last_clean":"2023-12-13T23:57:19.862106+0000","last_became_active":"2023-12-13T23:57:17.772029+0000","last_became_peered":"2023-12-13T23:57:17.772029+0000","last_unstale":"2023-12-13T23:57:19.862106+0000","last_undegraded":"2023-12-13T23:57:19.862106+0000","last_fullsized":"2023-12-13T23:57:19.862106+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:54:36.011059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530321+0000","last_change":"2023-12-13T23:57:16.997194+0000","last_active":"2023-12-13T23:57:19.530321+0000","last_peered":"2023-12-13T23:57:19.530321+0000","last_clean":"2023-12-13T23:57:19.530321+0000","last_became_active":"2023-12-13T23:57:16.996868+0000","last_became_peered":"2023-12-13T23:57:16.996868+0000","last_unstale":"2023-12-13T23:57:19.530321+0000","last_undegraded":"2023-12-13T23:57:19.530321+0000","last_fullsized":"2023-12-13T23:57:19.530321+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:58:31.762258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762329+0000","last_change":"2023-12-13T23:57:17.777392+0000","last_active":"2023-12-13T23:57:18.762329+0000","last_peered":"2023-12-13T23:57:18.762329+0000","last_clean":"2023-12-13T23:57:18.762329+0000","last_became_active":"2023-12-13T23:57:17.777188+0000","last_became_peered":"2023-12-13T23:57:17.777188+0000","last_unstale":"2023-12-13T23:57:18.762329+0000","last_undegraded":"2023-12-13T23:57:18.762329+0000","last_fullsized":"2023-12-13T23:57:18.762329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:42.851791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337037+0000","last_change":"2023-12-13T23:57:16.996782+0000","last_active":"2023-12-13T23:57:19.337037+0000","last_peered":"2023-12-13T23:57:19.337037+0000","last_clean":"2023-12-13T23:57:19.337037+0000","last_became_active":"2023-12-13T23:57:16.996495+0000","last_became_peered":"2023-12-13T23:57:16.996495+0000","last_unstale":"2023-12-13T23:57:19.337037+0000","last_undegraded":"2023-12-13T23:57:19.337037+0000","last_fullsized":"2023-12-13T23:57:19.337037+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:58:31.762258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760948+0000","last_change":"2023-12-13T23:57:17.770365+0000","last_active":"2023-12-13T23:57:18.760948+0000","last_peered":"2023-12-13T23:57:18.760948+0000","last_clean":"2023-12-13T23:57:18.760948+0000","last_became_active":"2023-12-13T23:57:17.770097+0000","last_became_peered":"2023-12-13T23:57:17.770097+0000","last_unstale":"2023-12-13T23:57:18.760948+0000","last_undegraded":"2023-12-13T23:57:18.760948+0000","last_fullsized":"2023-12-13T23:57:18.760948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:46.163512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762308+0000","last_change":"2023-12-13T23:57:16.996250+0000","last_active":"2023-12-13T23:57:18.762308+0000","last_peered":"2023-12-13T23:57:18.762308+0000","last_clean":"2023-12-13T23:57:18.762308+0000","last_became_active":"2023-12-13T23:57:16.995965+0000","last_became_peered":"2023-12-13T23:57:16.995965+0000","last_unstale":"2023-12-13T23:57:18.762308+0000","last_undegraded":"2023-12-13T23:57:18.762308+0000","last_fullsized":"2023-12-13T23:57:18.762308+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:31.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141477+0000","last_change":"2023-12-13T23:57:17.769249+0000","last_active":"2023-12-13T23:57:19.141477+0000","last_peered":"2023-12-13T23:57:19.141477+0000","last_clean":"2023-12-13T23:57:19.141477+0000","last_became_active":"2023-12-13T23:57:17.769002+0000","last_became_peered":"2023-12-13T23:57:17.769002+0000","last_unstale":"2023-12-13T23:57:19.141477+0000","last_undegraded":"2023-12-13T23:57:19.141477+0000","last_fullsized":"2023-12-13T23:57:19.141477+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:28:03.359766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761520+0000","last_change":"2023-12-13T23:57:17.003169+0000","last_active":"2023-12-13T23:57:18.761520+0000","last_peered":"2023-12-13T23:57:18.761520+0000","last_clean":"2023-12-13T23:57:18.761520+0000","last_became_active":"2023-12-13T23:57:17.002864+0000","last_became_peered":"2023-12-13T23:57:17.002864+0000","last_unstale":"2023-12-13T23:57:18.761520+0000","last_undegraded":"2023-12-13T23:57:18.761520+0000","last_fullsized":"2023-12-13T23:57:18.761520+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:24:50.583592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337375+0000","last_change":"2023-12-13T23:57:17.773985+0000","last_active":"2023-12-13T23:57:19.337375+0000","last_peered":"2023-12-13T23:57:19.337375+0000","last_clean":"2023-12-13T23:57:19.337375+0000","last_became_active":"2023-12-13T23:57:17.773312+0000","last_became_peered":"2023-12-13T23:57:17.773312+0000","last_unstale":"2023-12-13T23:57:19.337375+0000","last_undegraded":"2023-12-13T23:57:19.337375+0000","last_fullsized":"2023-12-13T23:57:19.337375+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:58:07.055202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762264+0000","last_change":"2023-12-13T23:57:17.001184+0000","last_active":"2023-12-13T23:57:18.762264+0000","last_peered":"2023-12-13T23:57:18.762264+0000","last_clean":"2023-12-13T23:57:18.762264+0000","last_became_active":"2023-12-13T23:57:17.000833+0000","last_became_peered":"2023-12-13T23:57:17.000833+0000","last_unstale":"2023-12-13T23:57:18.762264+0000","last_undegraded":"2023-12-13T23:57:18.762264+0000","last_fullsized":"2023-12-13T23:57:18.762264+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:52:28.107501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761381+0000","last_change":"2023-12-13T23:57:17.769101+0000","last_active":"2023-12-13T23:57:18.761381+0000","last_peered":"2023-12-13T23:57:18.761381+0000","last_clean":"2023-12-13T23:57:18.761381+0000","last_became_active":"2023-12-13T23:57:17.768823+0000","last_became_peered":"2023-12-13T23:57:17.768823+0000","last_unstale":"2023-12-13T23:57:18.761381+0000","last_undegraded":"2023-12-13T23:57:18.761381+0000","last_fullsized":"2023-12-13T23:57:18.761381+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:43:46.295106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862097+0000","last_change":"2023-12-13T23:57:17.003725+0000","last_active":"2023-12-13T23:57:19.862097+0000","last_peered":"2023-12-13T23:57:19.862097+0000","last_clean":"2023-12-13T23:57:19.862097+0000","last_became_active":"2023-12-13T23:57:17.003480+0000","last_became_peered":"2023-12-13T23:57:17.003480+0000","last_unstale":"2023-12-13T23:57:19.862097+0000","last_undegraded":"2023-12-13T23:57:19.862097+0000","last_fullsized":"2023-12-13T23:57:19.862097+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:41:32.437563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141511+0000","last_change":"2023-12-13T23:57:17.775275+0000","last_active":"2023-12-13T23:57:19.141511+0000","last_peered":"2023-12-13T23:57:19.141511+0000","last_clean":"2023-12-13T23:57:19.141511+0000","last_became_active":"2023-12-13T23:57:17.774973+0000","last_became_peered":"2023-12-13T23:57:17.774973+0000","last_unstale":"2023-12-13T23:57:19.141511+0000","last_undegraded":"2023-12-13T23:57:19.141511+0000","last_fullsized":"2023-12-13T23:57:19.141511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:55:00.239860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337291+0000","last_change":"2023-12-13T23:57:17.002204+0000","last_active":"2023-12-13T23:57:19.337291+0000","last_peered":"2023-12-13T23:57:19.337291+0000","last_clean":"2023-12-13T23:57:19.337291+0000","last_became_active":"2023-12-13T23:57:17.001540+0000","last_became_peered":"2023-12-13T23:57:17.001540+0000","last_unstale":"2023-12-13T23:57:19.337291+0000","last_undegraded":"2023-12-13T23:57:19.337291+0000","last_fullsized":"2023-12-13T23:57:19.337291+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:31:22.634702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337428+0000","last_change":"2023-12-13T23:57:17.770227+0000","last_active":"2023-12-13T23:57:19.337428+0000","last_peered":"2023-12-13T23:57:19.337428+0000","last_clean":"2023-12-13T23:57:19.337428+0000","last_became_active":"2023-12-13T23:57:17.769982+0000","last_became_peered":"2023-12-13T23:57:17.769982+0000","last_unstale":"2023-12-13T23:57:19.337428+0000","last_undegraded":"2023-12-13T23:57:19.337428+0000","last_fullsized":"2023-12-13T23:57:19.337428+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:54:00.786862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530392+0000","last_change":"2023-12-13T23:57:17.002250+0000","last_active":"2023-12-13T23:57:19.530392+0000","last_peered":"2023-12-13T23:57:19.530392+0000","last_clean":"2023-12-13T23:57:19.530392+0000","last_became_active":"2023-12-13T23:57:17.001944+0000","last_became_peered":"2023-12-13T23:57:17.001944+0000","last_unstale":"2023-12-13T23:57:19.530392+0000","last_undegraded":"2023-12-13T23:57:19.530392+0000","last_fullsized":"2023-12-13T23:57:19.530392+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:47:56.294820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861939+0000","last_change":"2023-12-13T23:57:17.770737+0000","last_active":"2023-12-13T23:57:19.861939+0000","last_peered":"2023-12-13T23:57:19.861939+0000","last_clean":"2023-12-13T23:57:19.861939+0000","last_became_active":"2023-12-13T23:57:17.770476+0000","last_became_peered":"2023-12-13T23:57:17.770476+0000","last_unstale":"2023-12-13T23:57:19.861939+0000","last_undegraded":"2023-12-13T23:57:19.861939+0000","last_fullsized":"2023-12-13T23:57:19.861939+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:36:22.300599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761479+0000","last_change":"2023-12-13T23:57:16.997540+0000","last_active":"2023-12-13T23:57:18.761479+0000","last_peered":"2023-12-13T23:57:18.761479+0000","last_clean":"2023-12-13T23:57:18.761479+0000","last_became_active":"2023-12-13T23:57:16.997015+0000","last_became_peered":"2023-12-13T23:57:16.997015+0000","last_unstale":"2023-12-13T23:57:18.761479+0000","last_undegraded":"2023-12-13T23:57:18.761479+0000","last_fullsized":"2023-12-13T23:57:18.761479+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337456+0000","last_change":"2023-12-13T23:57:17.773390+0000","last_active":"2023-12-13T23:57:19.337456+0000","last_peered":"2023-12-13T23:57:19.337456+0000","last_clean":"2023-12-13T23:57:19.337456+0000","last_became_active":"2023-12-13T23:57:17.773148+0000","last_became_peered":"2023-12-13T23:57:17.773148+0000","last_unstale":"2023-12-13T23:57:19.337456+0000","last_undegraded":"2023-12-13T23:57:19.337456+0000","last_fullsized":"2023-12-13T23:57:19.337456+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:49:50.405914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861702+0000","last_change":"2023-12-13T23:57:16.999816+0000","last_active":"2023-12-13T23:57:19.861702+0000","last_peered":"2023-12-13T23:57:19.861702+0000","last_clean":"2023-12-13T23:57:19.861702+0000","last_became_active":"2023-12-13T23:57:16.999571+0000","last_became_peered":"2023-12-13T23:57:16.999571+0000","last_unstale":"2023-12-13T23:57:19.861702+0000","last_undegraded":"2023-12-13T23:57:19.861702+0000","last_fullsized":"2023-12-13T23:57:19.861702+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:07:00.782128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861710+0000","last_change":"2023-12-13T23:57:17.770329+0000","last_active":"2023-12-13T23:57:19.861710+0000","last_peered":"2023-12-13T23:57:19.861710+0000","last_clean":"2023-12-13T23:57:19.861710+0000","last_became_active":"2023-12-13T23:57:17.770052+0000","last_became_peered":"2023-12-13T23:57:17.770052+0000","last_unstale":"2023-12-13T23:57:19.861710+0000","last_undegraded":"2023-12-13T23:57:19.861710+0000","last_fullsized":"2023-12-13T23:57:19.861710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:47:48.928358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337580+0000","last_change":"2023-12-13T23:57:16.997082+0000","last_active":"2023-12-13T23:57:19.337580+0000","last_peered":"2023-12-13T23:57:19.337580+0000","last_clean":"2023-12-13T23:57:19.337580+0000","last_became_active":"2023-12-13T23:57:16.996807+0000","last_became_peered":"2023-12-13T23:57:16.996807+0000","last_unstale":"2023-12-13T23:57:19.337580+0000","last_undegraded":"2023-12-13T23:57:19.337580+0000","last_fullsized":"2023-12-13T23:57:19.337580+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861413+0000","last_change":"2023-12-13T23:57:17.773144+0000","last_active":"2023-12-13T23:57:19.861413+0000","last_peered":"2023-12-13T23:57:19.861413+0000","last_clean":"2023-12-13T23:57:19.861413+0000","last_became_active":"2023-12-13T23:57:17.772875+0000","last_became_peered":"2023-12-13T23:57:17.772875+0000","last_unstale":"2023-12-13T23:57:19.861413+0000","last_undegraded":"2023-12-13T23:57:19.861413+0000","last_fullsized":"2023-12-13T23:57:19.861413+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:28:03.359766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760028+0000","last_change":"2023-12-13T23:57:16.997740+0000","last_active":"2023-12-13T23:57:18.760028+0000","last_peered":"2023-12-13T23:57:18.760028+0000","last_clean":"2023-12-13T23:57:18.760028+0000","last_became_active":"2023-12-13T23:57:16.997479+0000","last_became_peered":"2023-12-13T23:57:16.997479+0000","last_unstale":"2023-12-13T23:57:18.760028+0000","last_undegraded":"2023-12-13T23:57:18.760028+0000","last_fullsized":"2023-12-13T23:57:18.760028+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:55:16.107027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761454+0000","last_change":"2023-12-13T23:57:17.767817+0000","last_active":"2023-12-13T23:57:18.761454+0000","last_peered":"2023-12-13T23:57:18.761454+0000","last_clean":"2023-12-13T23:57:18.761454+0000","last_became_active":"2023-12-13T23:57:17.767439+0000","last_became_peered":"2023-12-13T23:57:17.767439+0000","last_unstale":"2023-12-13T23:57:18.761454+0000","last_undegraded":"2023-12-13T23:57:18.761454+0000","last_fullsized":"2023-12-13T23:57:18.761454+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:27:41.145805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530614+0000","last_change":"2023-12-13T23:57:16.996007+0000","last_active":"2023-12-13T23:57:19.530614+0000","last_peered":"2023-12-13T23:57:19.530614+0000","last_clean":"2023-12-13T23:57:19.530614+0000","last_became_active":"2023-12-13T23:57:16.995673+0000","last_became_peered":"2023-12-13T23:57:16.995673+0000","last_unstale":"2023-12-13T23:57:19.530614+0000","last_undegraded":"2023-12-13T23:57:19.530614+0000","last_fullsized":"2023-12-13T23:57:19.530614+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:21:23.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861525+0000","last_change":"2023-12-13T23:57:17.769082+0000","last_active":"2023-12-13T23:57:19.861525+0000","last_peered":"2023-12-13T23:57:19.861525+0000","last_clean":"2023-12-13T23:57:19.861525+0000","last_became_active":"2023-12-13T23:57:17.768807+0000","last_became_peered":"2023-12-13T23:57:17.768807+0000","last_unstale":"2023-12-13T23:57:19.861525+0000","last_undegraded":"2023-12-13T23:57:19.861525+0000","last_fullsized":"2023-12-13T23:57:19.861525+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:32.100913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530621+0000","last_change":"2023-12-13T23:57:16.997529+0000","last_active":"2023-12-13T23:57:19.530621+0000","last_peered":"2023-12-13T23:57:19.530621+0000","last_clean":"2023-12-13T23:57:19.530621+0000","last_became_active":"2023-12-13T23:57:16.997242+0000","last_became_peered":"2023-12-13T23:57:16.997242+0000","last_unstale":"2023-12-13T23:57:19.530621+0000","last_undegraded":"2023-12-13T23:57:19.530621+0000","last_fullsized":"2023-12-13T23:57:19.530621+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:35:56.043280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141646+0000","last_change":"2023-12-13T23:57:17.772911+0000","last_active":"2023-12-13T23:57:19.141646+0000","last_peered":"2023-12-13T23:57:19.141646+0000","last_clean":"2023-12-13T23:57:19.141646+0000","last_became_active":"2023-12-13T23:57:17.769997+0000","last_became_peered":"2023-12-13T23:57:17.769997+0000","last_unstale":"2023-12-13T23:57:19.141646+0000","last_undegraded":"2023-12-13T23:57:19.141646+0000","last_fullsized":"2023-12-13T23:57:19.141646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861571+0000","last_change":"2023-12-13T23:57:16.998017+0000","last_active":"2023-12-13T23:57:19.861571+0000","last_peered":"2023-12-13T23:57:19.861571+0000","last_clean":"2023-12-13T23:57:19.861571+0000","last_became_active":"2023-12-13T23:57:16.997742+0000","last_became_peered":"2023-12-13T23:57:16.997742+0000","last_unstale":"2023-12-13T23:57:19.861571+0000","last_undegraded":"2023-12-13T23:57:19.861571+0000","last_fullsized":"2023-12-13T23:57:19.861571+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530579+0000","last_change":"2023-12-13T23:57:17.769969+0000","last_active":"2023-12-13T23:57:19.530579+0000","last_peered":"2023-12-13T23:57:19.530579+0000","last_clean":"2023-12-13T23:57:19.530579+0000","last_became_active":"2023-12-13T23:57:17.769688+0000","last_became_peered":"2023-12-13T23:57:17.769688+0000","last_unstale":"2023-12-13T23:57:19.530579+0000","last_undegraded":"2023-12-13T23:57:19.530579+0000","last_fullsized":"2023-12-13T23:57:19.530579+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:06.342217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861543+0000","last_change":"2023-12-13T23:57:17.003382+0000","last_active":"2023-12-13T23:57:19.861543+0000","last_peered":"2023-12-13T23:57:19.861543+0000","last_clean":"2023-12-13T23:57:19.861543+0000","last_became_active":"2023-12-13T23:57:17.002947+0000","last_became_peered":"2023-12-13T23:57:17.002947+0000","last_unstale":"2023-12-13T23:57:19.861543+0000","last_undegraded":"2023-12-13T23:57:19.861543+0000","last_fullsized":"2023-12-13T23:57:19.861543+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:49:49.404005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337425+0000","last_change":"2023-12-13T23:57:17.769120+0000","last_active":"2023-12-13T23:57:19.337425+0000","last_peered":"2023-12-13T23:57:19.337425+0000","last_clean":"2023-12-13T23:57:19.337425+0000","last_became_active":"2023-12-13T23:57:17.768875+0000","last_became_peered":"2023-12-13T23:57:17.768875+0000","last_unstale":"2023-12-13T23:57:19.337425+0000","last_undegraded":"2023-12-13T23:57:19.337425+0000","last_fullsized":"2023-12-13T23:57:19.337425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:32:55.124768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337157+0000","last_change":"2023-12-13T23:57:16.998048+0000","last_active":"2023-12-13T23:57:19.337157+0000","last_peered":"2023-12-13T23:57:19.337157+0000","last_clean":"2023-12-13T23:57:19.337157+0000","last_became_active":"2023-12-13T23:57:16.997558+0000","last_became_peered":"2023-12-13T23:57:16.997558+0000","last_unstale":"2023-12-13T23:57:19.337157+0000","last_undegraded":"2023-12-13T23:57:19.337157+0000","last_fullsized":"2023-12-13T23:57:19.337157+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:45.161603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861864+0000","last_change":"2023-12-13T23:57:17.770499+0000","last_active":"2023-12-13T23:57:19.861864+0000","last_peered":"2023-12-13T23:57:19.861864+0000","last_clean":"2023-12-13T23:57:19.861864+0000","last_became_active":"2023-12-13T23:57:17.770243+0000","last_became_peered":"2023-12-13T23:57:17.770243+0000","last_unstale":"2023-12-13T23:57:19.861864+0000","last_undegraded":"2023-12-13T23:57:19.861864+0000","last_fullsized":"2023-12-13T23:57:19.861864+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.759768+0000","last_change":"2023-12-13T23:57:16.994643+0000","last_active":"2023-12-13T23:57:18.759768+0000","last_peered":"2023-12-13T23:57:18.759768+0000","last_clean":"2023-12-13T23:57:18.759768+0000","last_became_active":"2023-12-13T23:57:16.993858+0000","last_became_peered":"2023-12-13T23:57:16.993858+0000","last_unstale":"2023-12-13T23:57:18.759768+0000","last_undegraded":"2023-12-13T23:57:18.759768+0000","last_fullsized":"2023-12-13T23:57:18.759768+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:36:32.635221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141463+0000","last_change":"2023-12-13T23:57:17.773937+0000","last_active":"2023-12-13T23:57:19.141463+0000","last_peered":"2023-12-13T23:57:19.141463+0000","last_clean":"2023-12-13T23:57:19.141463+0000","last_became_active":"2023-12-13T23:57:17.773710+0000","last_became_peered":"2023-12-13T23:57:17.773710+0000","last_unstale":"2023-12-13T23:57:19.141463+0000","last_undegraded":"2023-12-13T23:57:19.141463+0000","last_fullsized":"2023-12-13T23:57:19.141463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:42.626381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530714+0000","last_change":"2023-12-13T23:57:17.002360+0000","last_active":"2023-12-13T23:57:19.530714+0000","last_peered":"2023-12-13T23:57:19.530714+0000","last_clean":"2023-12-13T23:57:19.530714+0000","last_became_active":"2023-12-13T23:57:17.001948+0000","last_became_peered":"2023-12-13T23:57:17.001948+0000","last_unstale":"2023-12-13T23:57:19.530714+0000","last_undegraded":"2023-12-13T23:57:19.530714+0000","last_fullsized":"2023-12-13T23:57:19.530714+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:37:47.093891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861853+0000","last_change":"2023-12-13T23:57:17.768960+0000","last_active":"2023-12-13T23:57:19.861853+0000","last_peered":"2023-12-13T23:57:19.861853+0000","last_clean":"2023-12-13T23:57:19.861853+0000","last_became_active":"2023-12-13T23:57:17.768692+0000","last_became_peered":"2023-12-13T23:57:17.768692+0000","last_unstale":"2023-12-13T23:57:19.861853+0000","last_undegraded":"2023-12-13T23:57:19.861853+0000","last_fullsized":"2023-12-13T23:57:19.861853+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762302+0000","last_change":"2023-12-13T23:57:16.999504+0000","last_active":"2023-12-13T23:57:18.762302+0000","last_peered":"2023-12-13T23:57:18.762302+0000","last_clean":"2023-12-13T23:57:18.762302+0000","last_became_active":"2023-12-13T23:57:16.999123+0000","last_became_peered":"2023-12-13T23:57:16.999123+0000","last_unstale":"2023-12-13T23:57:18.762302+0000","last_undegraded":"2023-12-13T23:57:18.762302+0000","last_fullsized":"2023-12-13T23:57:18.762302+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861717+0000","last_change":"2023-12-13T23:57:17.770897+0000","last_active":"2023-12-13T23:57:19.861717+0000","last_peered":"2023-12-13T23:57:19.861717+0000","last_clean":"2023-12-13T23:57:19.861717+0000","last_became_active":"2023-12-13T23:57:17.770120+0000","last_became_peered":"2023-12-13T23:57:17.770120+0000","last_unstale":"2023-12-13T23:57:19.861717+0000","last_undegraded":"2023-12-13T23:57:19.861717+0000","last_fullsized":"2023-12-13T23:57:19.861717+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:42:24.215630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337219+0000","last_change":"2023-12-13T23:57:17.001998+0000","last_active":"2023-12-13T23:57:19.337219+0000","last_peered":"2023-12-13T23:57:19.337219+0000","last_clean":"2023-12-13T23:57:19.337219+0000","last_became_active":"2023-12-13T23:57:17.001741+0000","last_became_peered":"2023-12-13T23:57:17.001741+0000","last_unstale":"2023-12-13T23:57:19.337219+0000","last_undegraded":"2023-12-13T23:57:19.337219+0000","last_fullsized":"2023-12-13T23:57:19.337219+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:26:34.277360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337245+0000","last_change":"2023-12-13T23:57:17.773847+0000","last_active":"2023-12-13T23:57:19.337245+0000","last_peered":"2023-12-13T23:57:19.337245+0000","last_clean":"2023-12-13T23:57:19.337245+0000","last_became_active":"2023-12-13T23:57:17.772054+0000","last_became_peered":"2023-12-13T23:57:17.772054+0000","last_unstale":"2023-12-13T23:57:19.337245+0000","last_undegraded":"2023-12-13T23:57:19.337245+0000","last_fullsized":"2023-12-13T23:57:19.337245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:14.954573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861737+0000","last_change":"2023-12-13T23:57:17.000763+0000","last_active":"2023-12-13T23:57:19.861737+0000","last_peered":"2023-12-13T23:57:19.861737+0000","last_clean":"2023-12-13T23:57:19.861737+0000","last_became_active":"2023-12-13T23:57:17.000524+0000","last_became_peered":"2023-12-13T23:57:17.000524+0000","last_unstale":"2023-12-13T23:57:19.861737+0000","last_undegraded":"2023-12-13T23:57:19.861737+0000","last_fullsized":"2023-12-13T23:57:19.861737+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:41.849881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141652+0000","last_change":"2023-12-13T23:57:17.775318+0000","last_active":"2023-12-13T23:57:19.141652+0000","last_peered":"2023-12-13T23:57:19.141652+0000","last_clean":"2023-12-13T23:57:19.141652+0000","last_became_active":"2023-12-13T23:57:17.775003+0000","last_became_peered":"2023-12-13T23:57:17.775003+0000","last_unstale":"2023-12-13T23:57:19.141652+0000","last_undegraded":"2023-12-13T23:57:19.141652+0000","last_fullsized":"2023-12-13T23:57:19.141652+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861594+0000","last_change":"2023-12-13T23:57:17.003199+0000","last_active":"2023-12-13T23:57:19.861594+0000","last_peered":"2023-12-13T23:57:19.861594+0000","last_clean":"2023-12-13T23:57:19.861594+0000","last_became_active":"2023-12-13T23:57:17.002578+0000","last_became_peered":"2023-12-13T23:57:17.002578+0000","last_unstale":"2023-12-13T23:57:19.861594+0000","last_undegraded":"2023-12-13T23:57:19.861594+0000","last_fullsized":"2023-12-13T23:57:19.861594+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:21:34.393111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337352+0000","last_change":"2023-12-13T23:57:17.771702+0000","last_active":"2023-12-13T23:57:19.337352+0000","last_peered":"2023-12-13T23:57:19.337352+0000","last_clean":"2023-12-13T23:57:19.337352+0000","last_became_active":"2023-12-13T23:57:17.771453+0000","last_became_peered":"2023-12-13T23:57:17.771453+0000","last_unstale":"2023-12-13T23:57:19.337352+0000","last_undegraded":"2023-12-13T23:57:19.337352+0000","last_fullsized":"2023-12-13T23:57:19.337352+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:09:58.638616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760992+0000","last_change":"2023-12-13T23:57:17.005167+0000","last_active":"2023-12-13T23:57:18.760992+0000","last_peered":"2023-12-13T23:57:18.760992+0000","last_clean":"2023-12-13T23:57:18.760992+0000","last_became_active":"2023-12-13T23:57:17.004864+0000","last_became_peered":"2023-12-13T23:57:17.004864+0000","last_unstale":"2023-12-13T23:57:18.760992+0000","last_undegraded":"2023-12-13T23:57:18.760992+0000","last_fullsized":"2023-12-13T23:57:18.760992+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:09:43.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862229+0000","last_change":"2023-12-13T23:57:17.772589+0000","last_active":"2023-12-13T23:57:19.862229+0000","last_peered":"2023-12-13T23:57:19.862229+0000","last_clean":"2023-12-13T23:57:19.862229+0000","last_became_active":"2023-12-13T23:57:17.772317+0000","last_became_peered":"2023-12-13T23:57:17.772317+0000","last_unstale":"2023-12-13T23:57:19.862229+0000","last_undegraded":"2023-12-13T23:57:19.862229+0000","last_fullsized":"2023-12-13T23:57:19.862229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:08:22.730122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141514+0000","last_change":"2023-12-13T23:57:17.001151+0000","last_active":"2023-12-13T23:57:19.141514+0000","last_peered":"2023-12-13T23:57:19.141514+0000","last_clean":"2023-12-13T23:57:19.141514+0000","last_became_active":"2023-12-13T23:57:17.000823+0000","last_became_peered":"2023-12-13T23:57:17.000823+0000","last_unstale":"2023-12-13T23:57:19.141514+0000","last_undegraded":"2023-12-13T23:57:19.141514+0000","last_fullsized":"2023-12-13T23:57:19.141514+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:26:34.277360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530358+0000","last_change":"2023-12-13T23:57:17.768951+0000","last_active":"2023-12-13T23:57:19.530358+0000","last_peered":"2023-12-13T23:57:19.530358+0000","last_clean":"2023-12-13T23:57:19.530358+0000","last_became_active":"2023-12-13T23:57:17.768649+0000","last_became_peered":"2023-12-13T23:57:17.768649+0000","last_unstale":"2023-12-13T23:57:19.530358+0000","last_undegraded":"2023-12-13T23:57:19.530358+0000","last_fullsized":"2023-12-13T23:57:19.530358+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:49:00.328074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337451+0000","last_change":"2023-12-13T23:57:17.000835+0000","last_active":"2023-12-13T23:57:19.337451+0000","last_peered":"2023-12-13T23:57:19.337451+0000","last_clean":"2023-12-13T23:57:19.337451+0000","last_became_active":"2023-12-13T23:57:16.997260+0000","last_became_peered":"2023-12-13T23:57:16.997260+0000","last_unstale":"2023-12-13T23:57:19.337451+0000","last_undegraded":"2023-12-13T23:57:19.337451+0000","last_fullsized":"2023-12-13T23:57:19.337451+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:31.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-13T23:57:17.758499+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:17.758499+0000","last_peered":"2023-12-13T23:57:17.758499+0000","last_clean":"2023-12-13T23:57:17.758499+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:17.758499+0000","last_undegraded":"2023-12-13T23:57:17.758499+0000","last_fullsized":"2023-12-13T23:57:17.758499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1196032,"data_stored":837070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.51200000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.496},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.40400000000000003}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56299999999999994}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52400000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57399999999999995}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.624}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.35399999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.376},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.496}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":246578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.432},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.45100000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.502}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58899999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.66500000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.71399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":598016,"data_stored":247160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.44700000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46300000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.311},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41999999999999998}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.374}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":248212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.25},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.37}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.39000000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46200000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.496},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":248190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.379}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.495},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55500000000000005}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":589824,"data_stored":240438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.437},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.40100000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.63800000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1183744,"data_stored":830702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.376}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.31900000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.505},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.56000000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.40799999999999997}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:20.937 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:21.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.072+0000 7fb850e3d640 1 Processor -- start 2023-12-13T23:57:21.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.072+0000 7fb850e3d640 1 -- start start 2023-12-13T23:57:21.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.073+0000 7fb850e3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 0x7fb84c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.073+0000 7fb850e3d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 0x7fb84c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.073+0000 7fb850e3d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb84c15e2a0 con 0x7fb84c15a4e0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.073+0000 7fb850e3d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb84c15b520 con 0x7fb84c15e730 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.073+0000 7fb850e3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c143870 con 0x7fb84c143490 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84ad9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb84c15a4e0 0x7fb84c15a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59192/0 (socket says 172.21.15.124:59192) 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84ad9d640 1 -- 172.21.15.124:0/1635365987 learned_addr learned my addr 172.21.15.124:0/1635365987 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84a59c640 1 --2- 172.21.15.124:0/1635365987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 0x7fb84c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb849d9b640 1 --2- 172.21.15.124:0/1635365987 >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 0x7fb84c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23734655 0 0) 0x7fb84c15e2a0 con 0x7fb84c15a4e0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb834005730 con 0x7fb84c15a4e0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155493925 0 0) 0x7fb834005730 con 0x7fb84c15a4e0 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84a59c640 1 -- 172.21.15.124:0/1635365987 >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 msgr2=0x7fb84c0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84a59c640 1 --2- 172.21.15.124:0/1635365987 >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 0x7fb84c0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84a59c640 1 -- 172.21.15.124:0/1635365987 >> v1:172.21.15.124:6789/0 conn(0x7fb84c15a4e0 legacy=0x7fb84c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb84a59c640 1 -- 172.21.15.124:0/1635365987 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c15b240 con 0x7fb84c143490 2023-12-13T23:57:21.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.074+0000 7fb849d9b640 1 --2- 172.21.15.124:0/1635365987 >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 0x7fb84c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.075+0000 7fb84a59c640 1 --2- 172.21.15.124:0/1635365987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 0x7fb84c0b0cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb84402b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc7061ab9a7a6044 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.075+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb83c001670 con 0x7fb84c143490 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.075+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84402e050 con 0x7fb84c143490 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.075+0000 7fb84959a640 1 -- 172.21.15.124:0/1635365987 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb844030730 con 0x7fb84c143490 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 -- 172.21.15.124:0/1635365987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 msgr2=0x7fb84c0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 --2- 172.21.15.124:0/1635365987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 0x7fb84c0b0cf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8440097a0 tx=0x7fb84402b790 comp rx=0 tx=0).stop 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 -- 172.21.15.124:0/1635365987 shutdown_connections 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 --2- 172.21.15.124:0/1635365987 >> v2:172.21.15.203:3300/0 conn(0x7fb84c15e730 0x7fb84c0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 --2- 172.21.15.124:0/1635365987 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c143490 0x7fb84c0b0cf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 -- 172.21.15.124:0/1635365987 >> 172.21.15.124:0/1635365987 conn(0x7fb84c06a4d0 msgr2=0x7fb84c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 -- 172.21.15.124:0/1635365987 shutdown_connections 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.076+0000 7fb850e3d640 1 -- 172.21.15.124:0/1635365987 wait complete. 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.077+0000 7fb850e3d640 1 Processor -- start 2023-12-13T23:57:21.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.077+0000 7fb850e3d640 1 -- start start 2023-12-13T23:57:21.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c15a4e0 0x7fb84c13bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb84c15e730 0x7fb84c13ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c13c0a0 con 0x7fb84c143490 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c13ff90 con 0x7fb84c15e730 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c140270 con 0x7fb84c15a4e0 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57092/0 (socket says 172.21.15.124:57092) 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 -- 172.21.15.124:0/2105181634 learned_addr learned my addr 172.21.15.124:0/2105181634 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84ad9d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb84c15e730 0x7fb84c13ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb849d9b640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c15a4e0 0x7fb84c13bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 -- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c15a4e0 msgr2=0x7fb84c13bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c15a4e0 0x7fb84c13bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 -- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb84c15e730 msgr2=0x7fb84c13ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb84c15e730 0x7fb84c13ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 -- 172.21.15.124:0/2105181634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb840009a70 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb84a59c640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb844000c00 tx=0x7fb8440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84403c880 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84403c9e0 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84403cd90 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.078+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb844009400 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.079+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c140830 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.079+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c069400 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.080+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb844039040 con 0x7fb84c143490 2023-12-13T23:57:21.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.080+0000 7fb83affd640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 0x7fb82c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.080+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb8440ab650 con 0x7fb84c143490 2023-12-13T23:57:21.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.080+0000 7fb849d9b640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 0x7fb82c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.081+0000 7fb849d9b640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 0x7fb82c06d6b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb84c13ed10 tx=0x7fb84000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.084+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb8440764b0 con 0x7fb84c143490 2023-12-13T23:57:21.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.249+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb84c15df30 con 0x7fb82c06b1f0 2023-12-13T23:57:21.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.257+0000 7fb83affd640 1 -- 172.21.15.124:0/2105181634 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+289603 (secure 0 0 0) 0x7fb84000f050 con 0x7fb82c06b1f0 2023-12-13T23:57:21.259 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:21.261 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 msgr2=0x7fb82c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 0x7fb82c06d6b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb84c13ed10 tx=0x7fb84000b040 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 msgr2=0x7fb84c159970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb844000c00 tx=0x7fb8440046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 shutdown_connections 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb82c06b1f0 0x7fb82c06d6b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb84c15e730 0x7fb84c13ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb84c15a4e0 0x7fb84c13bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 --2- 172.21.15.124:0/2105181634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb84c143490 0x7fb84c159970 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 >> 172.21.15.124:0/2105181634 conn(0x7fb84c06a4d0 msgr2=0x7fb84c15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 shutdown_connections 2023-12-13T23:57:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.260+0000 7fb850e3d640 1 -- 172.21.15.124:0/2105181634 wait complete. 2023-12-13T23:57:21.270 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2023-12-13T23:57:20.140742+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":155,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222036,"kb_used_data":5840,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752684,"statfs":{"total":773094113280,"available":772866748416,"internally_reserved":0,"allocated":5980160,"data_stored":3145580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":221236780},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.162839"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861819+0000","last_change":"2023-12-13T23:57:17.769091+0000","last_active":"2023-12-13T23:57:19.861819+0000","last_peered":"2023-12-13T23:57:19.861819+0000","last_clean":"2023-12-13T23:57:19.861819+0000","last_became_active":"2023-12-13T23:57:17.768834+0000","last_became_peered":"2023-12-13T23:57:17.768834+0000","last_unstale":"2023-12-13T23:57:19.861819+0000","last_undegraded":"2023-12-13T23:57:19.861819+0000","last_fullsized":"2023-12-13T23:57:19.861819+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762196+0000","last_change":"2023-12-13T23:57:17.767922+0000","last_active":"2023-12-13T23:57:18.762196+0000","last_peered":"2023-12-13T23:57:18.762196+0000","last_clean":"2023-12-13T23:57:18.762196+0000","last_became_active":"2023-12-13T23:57:17.767220+0000","last_became_peered":"2023-12-13T23:57:17.767220+0000","last_unstale":"2023-12-13T23:57:18.762196+0000","last_undegraded":"2023-12-13T23:57:18.762196+0000","last_fullsized":"2023-12-13T23:57:18.762196+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337273+0000","last_change":"2023-12-13T23:57:17.771889+0000","last_active":"2023-12-13T23:57:19.337273+0000","last_peered":"2023-12-13T23:57:19.337273+0000","last_clean":"2023-12-13T23:57:19.337273+0000","last_became_active":"2023-12-13T23:57:17.771650+0000","last_became_peered":"2023-12-13T23:57:17.771650+0000","last_unstale":"2023-12-13T23:57:19.337273+0000","last_undegraded":"2023-12-13T23:57:19.337273+0000","last_fullsized":"2023-12-13T23:57:19.337273+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:42.851791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762137+0000","last_change":"2023-12-13T23:57:17.773173+0000","last_active":"2023-12-13T23:57:18.762137+0000","last_peered":"2023-12-13T23:57:18.762137+0000","last_clean":"2023-12-13T23:57:18.762137+0000","last_became_active":"2023-12-13T23:57:17.772654+0000","last_became_peered":"2023-12-13T23:57:17.772654+0000","last_unstale":"2023-12-13T23:57:18.762137+0000","last_undegraded":"2023-12-13T23:57:18.762137+0000","last_fullsized":"2023-12-13T23:57:18.762137+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761242+0000","last_change":"2023-12-13T23:57:17.761851+0000","last_active":"2023-12-13T23:57:18.761242+0000","last_peered":"2023-12-13T23:57:18.761242+0000","last_clean":"2023-12-13T23:57:18.761242+0000","last_became_active":"2023-12-13T23:57:17.761449+0000","last_became_peered":"2023-12-13T23:57:17.761449+0000","last_unstale":"2023-12-13T23:57:18.761242+0000","last_undegraded":"2023-12-13T23:57:18.761242+0000","last_fullsized":"2023-12-13T23:57:18.761242+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:06.342217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861540+0000","last_change":"2023-12-13T23:57:17.772648+0000","last_active":"2023-12-13T23:57:19.861540+0000","last_peered":"2023-12-13T23:57:19.861540+0000","last_clean":"2023-12-13T23:57:19.861540+0000","last_became_active":"2023-12-13T23:57:17.772311+0000","last_became_peered":"2023-12-13T23:57:17.772311+0000","last_unstale":"2023-12-13T23:57:19.861540+0000","last_undegraded":"2023-12-13T23:57:19.861540+0000","last_fullsized":"2023-12-13T23:57:19.861540+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337050+0000","last_change":"2023-12-13T23:57:17.770329+0000","last_active":"2023-12-13T23:57:19.337050+0000","last_peered":"2023-12-13T23:57:19.337050+0000","last_clean":"2023-12-13T23:57:19.337050+0000","last_became_active":"2023-12-13T23:57:17.767466+0000","last_became_peered":"2023-12-13T23:57:17.767466+0000","last_unstale":"2023-12-13T23:57:19.337050+0000","last_undegraded":"2023-12-13T23:57:19.337050+0000","last_fullsized":"2023-12-13T23:57:19.337050+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861583+0000","last_change":"2023-12-13T23:57:17.780221+0000","last_active":"2023-12-13T23:57:19.861583+0000","last_peered":"2023-12-13T23:57:19.861583+0000","last_clean":"2023-12-13T23:57:19.861583+0000","last_became_active":"2023-12-13T23:57:17.779947+0000","last_became_peered":"2023-12-13T23:57:17.779947+0000","last_unstale":"2023-12-13T23:57:19.861583+0000","last_undegraded":"2023-12-13T23:57:19.861583+0000","last_fullsized":"2023-12-13T23:57:19.861583+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:20:25.103095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862215+0000","last_change":"2023-12-13T23:57:17.780203+0000","last_active":"2023-12-13T23:57:19.862215+0000","last_peered":"2023-12-13T23:57:19.862215+0000","last_clean":"2023-12-13T23:57:19.862215+0000","last_became_active":"2023-12-13T23:57:17.779908+0000","last_became_peered":"2023-12-13T23:57:17.779908+0000","last_unstale":"2023-12-13T23:57:19.862215+0000","last_undegraded":"2023-12-13T23:57:19.862215+0000","last_fullsized":"2023-12-13T23:57:19.862215+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:11:47.949638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861818+0000","last_change":"2023-12-13T23:57:17.769718+0000","last_active":"2023-12-13T23:57:19.861818+0000","last_peered":"2023-12-13T23:57:19.861818+0000","last_clean":"2023-12-13T23:57:19.861818+0000","last_became_active":"2023-12-13T23:57:17.769436+0000","last_became_peered":"2023-12-13T23:57:17.769436+0000","last_unstale":"2023-12-13T23:57:19.861818+0000","last_undegraded":"2023-12-13T23:57:19.861818+0000","last_fullsized":"2023-12-13T23:57:19.861818+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760066+0000","last_change":"2023-12-13T23:57:17.771798+0000","last_active":"2023-12-13T23:57:18.760066+0000","last_peered":"2023-12-13T23:57:18.760066+0000","last_clean":"2023-12-13T23:57:18.760066+0000","last_became_active":"2023-12-13T23:57:17.771543+0000","last_became_peered":"2023-12-13T23:57:17.771543+0000","last_unstale":"2023-12-13T23:57:18.760066+0000","last_undegraded":"2023-12-13T23:57:18.760066+0000","last_fullsized":"2023-12-13T23:57:18.760066+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:40:05.961525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862100+0000","last_change":"2023-12-13T23:57:17.769855+0000","last_active":"2023-12-13T23:57:19.862100+0000","last_peered":"2023-12-13T23:57:19.862100+0000","last_clean":"2023-12-13T23:57:19.862100+0000","last_became_active":"2023-12-13T23:57:17.769499+0000","last_became_peered":"2023-12-13T23:57:17.769499+0000","last_unstale":"2023-12-13T23:57:19.862100+0000","last_undegraded":"2023-12-13T23:57:19.862100+0000","last_fullsized":"2023-12-13T23:57:19.862100+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:37:36.347678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861473+0000","last_change":"2023-12-13T23:57:17.769967+0000","last_active":"2023-12-13T23:57:19.861473+0000","last_peered":"2023-12-13T23:57:19.861473+0000","last_clean":"2023-12-13T23:57:19.861473+0000","last_became_active":"2023-12-13T23:57:17.769443+0000","last_became_peered":"2023-12-13T23:57:17.769443+0000","last_unstale":"2023-12-13T23:57:19.861473+0000","last_undegraded":"2023-12-13T23:57:19.861473+0000","last_fullsized":"2023-12-13T23:57:19.861473+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:55:00.239860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337219+0000","last_change":"2023-12-13T23:57:17.774073+0000","last_active":"2023-12-13T23:57:19.337219+0000","last_peered":"2023-12-13T23:57:19.337219+0000","last_clean":"2023-12-13T23:57:19.337219+0000","last_became_active":"2023-12-13T23:57:17.768211+0000","last_became_peered":"2023-12-13T23:57:17.768211+0000","last_unstale":"2023-12-13T23:57:19.337219+0000","last_undegraded":"2023-12-13T23:57:19.337219+0000","last_fullsized":"2023-12-13T23:57:19.337219+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761384+0000","last_change":"2023-12-13T23:57:17.770058+0000","last_active":"2023-12-13T23:57:18.761384+0000","last_peered":"2023-12-13T23:57:18.761384+0000","last_clean":"2023-12-13T23:57:18.761384+0000","last_became_active":"2023-12-13T23:57:17.769678+0000","last_became_peered":"2023-12-13T23:57:17.769678+0000","last_unstale":"2023-12-13T23:57:18.761384+0000","last_undegraded":"2023-12-13T23:57:18.761384+0000","last_fullsized":"2023-12-13T23:57:18.761384+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861379+0000","last_change":"2023-12-13T23:57:17.771706+0000","last_active":"2023-12-13T23:57:19.861379+0000","last_peered":"2023-12-13T23:57:19.861379+0000","last_clean":"2023-12-13T23:57:19.861379+0000","last_became_active":"2023-12-13T23:57:17.771466+0000","last_became_peered":"2023-12-13T23:57:17.771466+0000","last_unstale":"2023-12-13T23:57:19.861379+0000","last_undegraded":"2023-12-13T23:57:19.861379+0000","last_fullsized":"2023-12-13T23:57:19.861379+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:42.626381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337534+0000","last_change":"2023-12-13T23:57:17.772457+0000","last_active":"2023-12-13T23:57:19.337534+0000","last_peered":"2023-12-13T23:57:19.337534+0000","last_clean":"2023-12-13T23:57:19.337534+0000","last_became_active":"2023-12-13T23:57:17.772211+0000","last_became_peered":"2023-12-13T23:57:17.772211+0000","last_unstale":"2023-12-13T23:57:19.337534+0000","last_undegraded":"2023-12-13T23:57:19.337534+0000","last_fullsized":"2023-12-13T23:57:19.337534+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:07:01.784038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762266+0000","last_change":"2023-12-13T23:57:17.771231+0000","last_active":"2023-12-13T23:57:18.762266+0000","last_peered":"2023-12-13T23:57:18.762266+0000","last_clean":"2023-12-13T23:57:18.762266+0000","last_became_active":"2023-12-13T23:57:17.770956+0000","last_became_peered":"2023-12-13T23:57:17.770956+0000","last_unstale":"2023-12-13T23:57:18.762266+0000","last_undegraded":"2023-12-13T23:57:18.762266+0000","last_fullsized":"2023-12-13T23:57:18.762266+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:14.954573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337335+0000","last_change":"2023-12-13T23:57:17.775732+0000","last_active":"2023-12-13T23:57:19.337335+0000","last_peered":"2023-12-13T23:57:19.337335+0000","last_clean":"2023-12-13T23:57:19.337335+0000","last_became_active":"2023-12-13T23:57:17.775357+0000","last_became_peered":"2023-12-13T23:57:17.775357+0000","last_unstale":"2023-12-13T23:57:19.337335+0000","last_undegraded":"2023-12-13T23:57:19.337335+0000","last_fullsized":"2023-12-13T23:57:19.337335+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:41:33.439472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530542+0000","last_change":"2023-12-13T23:57:17.770778+0000","last_active":"2023-12-13T23:57:19.530542+0000","last_peered":"2023-12-13T23:57:19.530542+0000","last_clean":"2023-12-13T23:57:19.530542+0000","last_became_active":"2023-12-13T23:57:17.770323+0000","last_became_peered":"2023-12-13T23:57:17.770323+0000","last_unstale":"2023-12-13T23:57:19.530542+0000","last_undegraded":"2023-12-13T23:57:19.530542+0000","last_fullsized":"2023-12-13T23:57:19.530542+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:46.163512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530460+0000","last_change":"2023-12-13T23:57:17.770750+0000","last_active":"2023-12-13T23:57:19.530460+0000","last_peered":"2023-12-13T23:57:19.530460+0000","last_clean":"2023-12-13T23:57:19.530460+0000","last_became_active":"2023-12-13T23:57:17.770470+0000","last_became_peered":"2023-12-13T23:57:17.770470+0000","last_unstale":"2023-12-13T23:57:19.530460+0000","last_undegraded":"2023-12-13T23:57:19.530460+0000","last_fullsized":"2023-12-13T23:57:19.530460+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:24:51.585501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861444+0000","last_change":"2023-12-13T23:57:17.780310+0000","last_active":"2023-12-13T23:57:19.861444+0000","last_peered":"2023-12-13T23:57:19.861444+0000","last_clean":"2023-12-13T23:57:19.861444+0000","last_became_active":"2023-12-13T23:57:17.780111+0000","last_became_peered":"2023-12-13T23:57:17.780111+0000","last_unstale":"2023-12-13T23:57:19.861444+0000","last_undegraded":"2023-12-13T23:57:19.861444+0000","last_fullsized":"2023-12-13T23:57:19.861444+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T08:17:24.759895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.759876+0000","last_change":"2023-12-13T23:57:17.771057+0000","last_active":"2023-12-13T23:57:18.759876+0000","last_peered":"2023-12-13T23:57:18.759876+0000","last_clean":"2023-12-13T23:57:18.759876+0000","last_became_active":"2023-12-13T23:57:17.770813+0000","last_became_peered":"2023-12-13T23:57:17.770813+0000","last_unstale":"2023-12-13T23:57:18.759876+0000","last_undegraded":"2023-12-13T23:57:18.759876+0000","last_fullsized":"2023-12-13T23:57:18.759876+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:05:52.770820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337013+0000","last_change":"2023-12-13T23:57:17.767485+0000","last_active":"2023-12-13T23:57:19.337013+0000","last_peered":"2023-12-13T23:57:19.337013+0000","last_clean":"2023-12-13T23:57:19.337013+0000","last_became_active":"2023-12-13T23:57:17.767236+0000","last_became_peered":"2023-12-13T23:57:17.767236+0000","last_unstale":"2023-12-13T23:57:19.337013+0000","last_undegraded":"2023-12-13T23:57:19.337013+0000","last_fullsized":"2023-12-13T23:57:19.337013+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337535+0000","last_change":"2023-12-13T23:57:17.771759+0000","last_active":"2023-12-13T23:57:19.337535+0000","last_peered":"2023-12-13T23:57:19.337535+0000","last_clean":"2023-12-13T23:57:19.337535+0000","last_became_active":"2023-12-13T23:57:17.771490+0000","last_became_peered":"2023-12-13T23:57:17.771490+0000","last_unstale":"2023-12-13T23:57:19.337535+0000","last_undegraded":"2023-12-13T23:57:19.337535+0000","last_fullsized":"2023-12-13T23:57:19.337535+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:21:35.395021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861939+0000","last_change":"2023-12-13T23:57:17.772866+0000","last_active":"2023-12-13T23:57:19.861939+0000","last_peered":"2023-12-13T23:57:19.861939+0000","last_clean":"2023-12-13T23:57:19.861939+0000","last_became_active":"2023-12-13T23:57:17.772350+0000","last_became_peered":"2023-12-13T23:57:17.772350+0000","last_unstale":"2023-12-13T23:57:19.861939+0000","last_undegraded":"2023-12-13T23:57:19.861939+0000","last_fullsized":"2023-12-13T23:57:19.861939+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:54:07.211775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862042+0000","last_change":"2023-12-13T23:57:17.777353+0000","last_active":"2023-12-13T23:57:19.862042+0000","last_peered":"2023-12-13T23:57:19.862042+0000","last_clean":"2023-12-13T23:57:19.862042+0000","last_became_active":"2023-12-13T23:57:17.777104+0000","last_became_peered":"2023-12-13T23:57:17.777104+0000","last_unstale":"2023-12-13T23:57:19.862042+0000","last_undegraded":"2023-12-13T23:57:19.862042+0000","last_fullsized":"2023-12-13T23:57:19.862042+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530616+0000","last_change":"2023-12-13T23:57:17.771415+0000","last_active":"2023-12-13T23:57:19.530616+0000","last_peered":"2023-12-13T23:57:19.530616+0000","last_clean":"2023-12-13T23:57:19.530616+0000","last_became_active":"2023-12-13T23:57:17.771091+0000","last_became_peered":"2023-12-13T23:57:17.771091+0000","last_unstale":"2023-12-13T23:57:19.530616+0000","last_undegraded":"2023-12-13T23:57:19.530616+0000","last_fullsized":"2023-12-13T23:57:19.530616+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:10:40.296399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337330+0000","last_change":"2023-12-13T23:57:17.768872+0000","last_active":"2023-12-13T23:57:19.337330+0000","last_peered":"2023-12-13T23:57:19.337330+0000","last_clean":"2023-12-13T23:57:19.337330+0000","last_became_active":"2023-12-13T23:57:17.768630+0000","last_became_peered":"2023-12-13T23:57:17.768630+0000","last_unstale":"2023-12-13T23:57:19.337330+0000","last_undegraded":"2023-12-13T23:57:19.337330+0000","last_fullsized":"2023-12-13T23:57:19.337330+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:16:08.664883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761749+0000","last_change":"2023-12-13T23:57:17.769015+0000","last_active":"2023-12-13T23:57:18.761749+0000","last_peered":"2023-12-13T23:57:18.761749+0000","last_clean":"2023-12-13T23:57:18.761749+0000","last_became_active":"2023-12-13T23:57:17.768743+0000","last_became_peered":"2023-12-13T23:57:17.768743+0000","last_unstale":"2023-12-13T23:57:18.761749+0000","last_undegraded":"2023-12-13T23:57:18.761749+0000","last_fullsized":"2023-12-13T23:57:18.761749+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337118+0000","last_change":"2023-12-13T23:57:17.768464+0000","last_active":"2023-12-13T23:57:19.337118+0000","last_peered":"2023-12-13T23:57:19.337118+0000","last_clean":"2023-12-13T23:57:19.337118+0000","last_became_active":"2023-12-13T23:57:17.767146+0000","last_became_peered":"2023-12-13T23:57:17.767146+0000","last_unstale":"2023-12-13T23:57:19.337118+0000","last_undegraded":"2023-12-13T23:57:19.337118+0000","last_fullsized":"2023-12-13T23:57:19.337118+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141425+0000","last_change":"2023-12-13T23:57:17.774634+0000","last_active":"2023-12-13T23:57:19.141425+0000","last_peered":"2023-12-13T23:57:19.141425+0000","last_clean":"2023-12-13T23:57:19.141425+0000","last_became_active":"2023-12-13T23:57:17.774342+0000","last_became_peered":"2023-12-13T23:57:17.774342+0000","last_unstale":"2023-12-13T23:57:19.141425+0000","last_undegraded":"2023-12-13T23:57:19.141425+0000","last_fullsized":"2023-12-13T23:57:19.141425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:45.776626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761890+0000","last_change":"2023-12-13T23:57:17.771633+0000","last_active":"2023-12-13T23:57:18.761890+0000","last_peered":"2023-12-13T23:57:18.761890+0000","last_clean":"2023-12-13T23:57:18.761890+0000","last_became_active":"2023-12-13T23:57:17.771365+0000","last_became_peered":"2023-12-13T23:57:17.771365+0000","last_unstale":"2023-12-13T23:57:18.761890+0000","last_undegraded":"2023-12-13T23:57:18.761890+0000","last_fullsized":"2023-12-13T23:57:18.761890+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:00:26.311307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"49'4","reported_seq":16,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530730+0000","last_change":"2023-12-13T23:57:16.996579+0000","last_active":"2023-12-13T23:57:19.530730+0000","last_peered":"2023-12-13T23:57:19.530730+0000","last_clean":"2023-12-13T23:57:19.530730+0000","last_became_active":"2023-12-13T23:57:16.996282+0000","last_became_peered":"2023-12-13T23:57:16.996282+0000","last_unstale":"2023-12-13T23:57:19.530730+0000","last_undegraded":"2023-12-13T23:57:19.530730+0000","last_fullsized":"2023-12-13T23:57:19.530730+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:22:10.185846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141544+0000","last_change":"2023-12-13T23:57:17.775233+0000","last_active":"2023-12-13T23:57:19.141544+0000","last_peered":"2023-12-13T23:57:19.141544+0000","last_clean":"2023-12-13T23:57:19.141544+0000","last_became_active":"2023-12-13T23:57:17.774947+0000","last_became_peered":"2023-12-13T23:57:17.774947+0000","last_unstale":"2023-12-13T23:57:19.141544+0000","last_undegraded":"2023-12-13T23:57:19.141544+0000","last_fullsized":"2023-12-13T23:57:19.141544+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:32.100913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861883+0000","last_change":"2023-12-13T23:57:16.997782+0000","last_active":"2023-12-13T23:57:19.861883+0000","last_peered":"2023-12-13T23:57:19.861883+0000","last_clean":"2023-12-13T23:57:19.861883+0000","last_became_active":"2023-12-13T23:57:16.997203+0000","last_became_peered":"2023-12-13T23:57:16.997203+0000","last_unstale":"2023-12-13T23:57:19.861883+0000","last_undegraded":"2023-12-13T23:57:19.861883+0000","last_fullsized":"2023-12-13T23:57:19.861883+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:27:40.143895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530463+0000","last_change":"2023-12-13T23:57:17.770837+0000","last_active":"2023-12-13T23:57:19.530463+0000","last_peered":"2023-12-13T23:57:19.530463+0000","last_clean":"2023-12-13T23:57:19.530463+0000","last_became_active":"2023-12-13T23:57:17.770461+0000","last_became_peered":"2023-12-13T23:57:17.770461+0000","last_unstale":"2023-12-13T23:57:19.530463+0000","last_undegraded":"2023-12-13T23:57:19.530463+0000","last_fullsized":"2023-12-13T23:57:19.530463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:31:23.636612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861859+0000","last_change":"2023-12-13T23:57:16.991830+0000","last_active":"2023-12-13T23:57:19.861859+0000","last_peered":"2023-12-13T23:57:19.861859+0000","last_clean":"2023-12-13T23:57:19.861859+0000","last_became_active":"2023-12-13T23:57:16.991558+0000","last_became_peered":"2023-12-13T23:57:16.991558+0000","last_unstale":"2023-12-13T23:57:19.861859+0000","last_undegraded":"2023-12-13T23:57:19.861859+0000","last_fullsized":"2023-12-13T23:57:19.861859+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861654+0000","last_change":"2023-12-13T23:57:17.772416+0000","last_active":"2023-12-13T23:57:19.861654+0000","last_peered":"2023-12-13T23:57:19.861654+0000","last_clean":"2023-12-13T23:57:19.861654+0000","last_became_active":"2023-12-13T23:57:17.772173+0000","last_became_peered":"2023-12-13T23:57:17.772173+0000","last_unstale":"2023-12-13T23:57:19.861654+0000","last_undegraded":"2023-12-13T23:57:19.861654+0000","last_fullsized":"2023-12-13T23:57:19.861654+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:24:48.909848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141617+0000","last_change":"2023-12-13T23:57:17.001547+0000","last_active":"2023-12-13T23:57:19.141617+0000","last_peered":"2023-12-13T23:57:19.141617+0000","last_clean":"2023-12-13T23:57:19.141617+0000","last_became_active":"2023-12-13T23:57:17.001344+0000","last_became_peered":"2023-12-13T23:57:17.001344+0000","last_unstale":"2023-12-13T23:57:19.141617+0000","last_undegraded":"2023-12-13T23:57:19.141617+0000","last_fullsized":"2023-12-13T23:57:19.141617+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761257+0000","last_change":"2023-12-13T23:57:17.768218+0000","last_active":"2023-12-13T23:57:18.761257+0000","last_peered":"2023-12-13T23:57:18.761257+0000","last_clean":"2023-12-13T23:57:18.761257+0000","last_became_active":"2023-12-13T23:57:17.767959+0000","last_became_peered":"2023-12-13T23:57:17.767959+0000","last_unstale":"2023-12-13T23:57:18.761257+0000","last_undegraded":"2023-12-13T23:57:18.761257+0000","last_fullsized":"2023-12-13T23:57:18.761257+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:10:40.296399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337167+0000","last_change":"2023-12-13T23:57:17.002065+0000","last_active":"2023-12-13T23:57:19.337167+0000","last_peered":"2023-12-13T23:57:19.337167+0000","last_clean":"2023-12-13T23:57:19.337167+0000","last_became_active":"2023-12-13T23:57:17.001815+0000","last_became_peered":"2023-12-13T23:57:17.001815+0000","last_unstale":"2023-12-13T23:57:19.337167+0000","last_undegraded":"2023-12-13T23:57:19.337167+0000","last_fullsized":"2023-12-13T23:57:19.337167+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:05.340307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762203+0000","last_change":"2023-12-13T23:57:17.768443+0000","last_active":"2023-12-13T23:57:18.762203+0000","last_peered":"2023-12-13T23:57:18.762203+0000","last_clean":"2023-12-13T23:57:18.762203+0000","last_became_active":"2023-12-13T23:57:17.768139+0000","last_became_peered":"2023-12-13T23:57:17.768139+0000","last_unstale":"2023-12-13T23:57:18.762203+0000","last_undegraded":"2023-12-13T23:57:18.762203+0000","last_fullsized":"2023-12-13T23:57:18.762203+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:37:28.597056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761177+0000","last_change":"2023-12-13T23:57:16.999010+0000","last_active":"2023-12-13T23:57:18.761177+0000","last_peered":"2023-12-13T23:57:18.761177+0000","last_clean":"2023-12-13T23:57:18.761177+0000","last_became_active":"2023-12-13T23:57:16.998688+0000","last_became_peered":"2023-12-13T23:57:16.998688+0000","last_unstale":"2023-12-13T23:57:18.761177+0000","last_undegraded":"2023-12-13T23:57:18.761177+0000","last_fullsized":"2023-12-13T23:57:18.761177+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:39:14.365772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862106+0000","last_change":"2023-12-13T23:57:17.772583+0000","last_active":"2023-12-13T23:57:19.862106+0000","last_peered":"2023-12-13T23:57:19.862106+0000","last_clean":"2023-12-13T23:57:19.862106+0000","last_became_active":"2023-12-13T23:57:17.772029+0000","last_became_peered":"2023-12-13T23:57:17.772029+0000","last_unstale":"2023-12-13T23:57:19.862106+0000","last_undegraded":"2023-12-13T23:57:19.862106+0000","last_fullsized":"2023-12-13T23:57:19.862106+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:54:36.011059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530321+0000","last_change":"2023-12-13T23:57:16.997194+0000","last_active":"2023-12-13T23:57:19.530321+0000","last_peered":"2023-12-13T23:57:19.530321+0000","last_clean":"2023-12-13T23:57:19.530321+0000","last_became_active":"2023-12-13T23:57:16.996868+0000","last_became_peered":"2023-12-13T23:57:16.996868+0000","last_unstale":"2023-12-13T23:57:19.530321+0000","last_undegraded":"2023-12-13T23:57:19.530321+0000","last_fullsized":"2023-12-13T23:57:19.530321+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:58:31.762258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762329+0000","last_change":"2023-12-13T23:57:17.777392+0000","last_active":"2023-12-13T23:57:18.762329+0000","last_peered":"2023-12-13T23:57:18.762329+0000","last_clean":"2023-12-13T23:57:18.762329+0000","last_became_active":"2023-12-13T23:57:17.777188+0000","last_became_peered":"2023-12-13T23:57:17.777188+0000","last_unstale":"2023-12-13T23:57:18.762329+0000","last_undegraded":"2023-12-13T23:57:18.762329+0000","last_fullsized":"2023-12-13T23:57:18.762329+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:42.851791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337037+0000","last_change":"2023-12-13T23:57:16.996782+0000","last_active":"2023-12-13T23:57:19.337037+0000","last_peered":"2023-12-13T23:57:19.337037+0000","last_clean":"2023-12-13T23:57:19.337037+0000","last_became_active":"2023-12-13T23:57:16.996495+0000","last_became_peered":"2023-12-13T23:57:16.996495+0000","last_unstale":"2023-12-13T23:57:19.337037+0000","last_undegraded":"2023-12-13T23:57:19.337037+0000","last_fullsized":"2023-12-13T23:57:19.337037+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:58:31.762258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760948+0000","last_change":"2023-12-13T23:57:17.770365+0000","last_active":"2023-12-13T23:57:18.760948+0000","last_peered":"2023-12-13T23:57:18.760948+0000","last_clean":"2023-12-13T23:57:18.760948+0000","last_became_active":"2023-12-13T23:57:17.770097+0000","last_became_peered":"2023-12-13T23:57:17.770097+0000","last_unstale":"2023-12-13T23:57:18.760948+0000","last_undegraded":"2023-12-13T23:57:18.760948+0000","last_fullsized":"2023-12-13T23:57:18.760948+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:46.163512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762308+0000","last_change":"2023-12-13T23:57:16.996250+0000","last_active":"2023-12-13T23:57:18.762308+0000","last_peered":"2023-12-13T23:57:18.762308+0000","last_clean":"2023-12-13T23:57:18.762308+0000","last_became_active":"2023-12-13T23:57:16.995965+0000","last_became_peered":"2023-12-13T23:57:16.995965+0000","last_unstale":"2023-12-13T23:57:18.762308+0000","last_undegraded":"2023-12-13T23:57:18.762308+0000","last_fullsized":"2023-12-13T23:57:18.762308+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:31.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141477+0000","last_change":"2023-12-13T23:57:17.769249+0000","last_active":"2023-12-13T23:57:19.141477+0000","last_peered":"2023-12-13T23:57:19.141477+0000","last_clean":"2023-12-13T23:57:19.141477+0000","last_became_active":"2023-12-13T23:57:17.769002+0000","last_became_peered":"2023-12-13T23:57:17.769002+0000","last_unstale":"2023-12-13T23:57:19.141477+0000","last_undegraded":"2023-12-13T23:57:19.141477+0000","last_fullsized":"2023-12-13T23:57:19.141477+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:28:03.359766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761520+0000","last_change":"2023-12-13T23:57:17.003169+0000","last_active":"2023-12-13T23:57:18.761520+0000","last_peered":"2023-12-13T23:57:18.761520+0000","last_clean":"2023-12-13T23:57:18.761520+0000","last_became_active":"2023-12-13T23:57:17.002864+0000","last_became_peered":"2023-12-13T23:57:17.002864+0000","last_unstale":"2023-12-13T23:57:18.761520+0000","last_undegraded":"2023-12-13T23:57:18.761520+0000","last_fullsized":"2023-12-13T23:57:18.761520+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:24:50.583592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337375+0000","last_change":"2023-12-13T23:57:17.773985+0000","last_active":"2023-12-13T23:57:19.337375+0000","last_peered":"2023-12-13T23:57:19.337375+0000","last_clean":"2023-12-13T23:57:19.337375+0000","last_became_active":"2023-12-13T23:57:17.773312+0000","last_became_peered":"2023-12-13T23:57:17.773312+0000","last_unstale":"2023-12-13T23:57:19.337375+0000","last_undegraded":"2023-12-13T23:57:19.337375+0000","last_fullsized":"2023-12-13T23:57:19.337375+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:58:07.055202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"49'3","reported_seq":15,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762264+0000","last_change":"2023-12-13T23:57:17.001184+0000","last_active":"2023-12-13T23:57:18.762264+0000","last_peered":"2023-12-13T23:57:18.762264+0000","last_clean":"2023-12-13T23:57:18.762264+0000","last_became_active":"2023-12-13T23:57:17.000833+0000","last_became_peered":"2023-12-13T23:57:17.000833+0000","last_unstale":"2023-12-13T23:57:18.762264+0000","last_undegraded":"2023-12-13T23:57:18.762264+0000","last_fullsized":"2023-12-13T23:57:18.762264+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:52:28.107501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761381+0000","last_change":"2023-12-13T23:57:17.769101+0000","last_active":"2023-12-13T23:57:18.761381+0000","last_peered":"2023-12-13T23:57:18.761381+0000","last_clean":"2023-12-13T23:57:18.761381+0000","last_became_active":"2023-12-13T23:57:17.768823+0000","last_became_peered":"2023-12-13T23:57:17.768823+0000","last_unstale":"2023-12-13T23:57:18.761381+0000","last_undegraded":"2023-12-13T23:57:18.761381+0000","last_fullsized":"2023-12-13T23:57:18.761381+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:43:46.295106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"49'6","reported_seq":18,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862097+0000","last_change":"2023-12-13T23:57:17.003725+0000","last_active":"2023-12-13T23:57:19.862097+0000","last_peered":"2023-12-13T23:57:19.862097+0000","last_clean":"2023-12-13T23:57:19.862097+0000","last_became_active":"2023-12-13T23:57:17.003480+0000","last_became_peered":"2023-12-13T23:57:17.003480+0000","last_unstale":"2023-12-13T23:57:19.862097+0000","last_undegraded":"2023-12-13T23:57:19.862097+0000","last_fullsized":"2023-12-13T23:57:19.862097+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T04:41:32.437563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141511+0000","last_change":"2023-12-13T23:57:17.775275+0000","last_active":"2023-12-13T23:57:19.141511+0000","last_peered":"2023-12-13T23:57:19.141511+0000","last_clean":"2023-12-13T23:57:19.141511+0000","last_became_active":"2023-12-13T23:57:17.774973+0000","last_became_peered":"2023-12-13T23:57:17.774973+0000","last_unstale":"2023-12-13T23:57:19.141511+0000","last_undegraded":"2023-12-13T23:57:19.141511+0000","last_fullsized":"2023-12-13T23:57:19.141511+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:55:00.239860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337291+0000","last_change":"2023-12-13T23:57:17.002204+0000","last_active":"2023-12-13T23:57:19.337291+0000","last_peered":"2023-12-13T23:57:19.337291+0000","last_clean":"2023-12-13T23:57:19.337291+0000","last_became_active":"2023-12-13T23:57:17.001540+0000","last_became_peered":"2023-12-13T23:57:17.001540+0000","last_unstale":"2023-12-13T23:57:19.337291+0000","last_undegraded":"2023-12-13T23:57:19.337291+0000","last_fullsized":"2023-12-13T23:57:19.337291+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:31:22.634702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337428+0000","last_change":"2023-12-13T23:57:17.770227+0000","last_active":"2023-12-13T23:57:19.337428+0000","last_peered":"2023-12-13T23:57:19.337428+0000","last_clean":"2023-12-13T23:57:19.337428+0000","last_became_active":"2023-12-13T23:57:17.769982+0000","last_became_peered":"2023-12-13T23:57:17.769982+0000","last_unstale":"2023-12-13T23:57:19.337428+0000","last_undegraded":"2023-12-13T23:57:19.337428+0000","last_fullsized":"2023-12-13T23:57:19.337428+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:54:00.786862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530392+0000","last_change":"2023-12-13T23:57:17.002250+0000","last_active":"2023-12-13T23:57:19.530392+0000","last_peered":"2023-12-13T23:57:19.530392+0000","last_clean":"2023-12-13T23:57:19.530392+0000","last_became_active":"2023-12-13T23:57:17.001944+0000","last_became_peered":"2023-12-13T23:57:17.001944+0000","last_unstale":"2023-12-13T23:57:19.530392+0000","last_undegraded":"2023-12-13T23:57:19.530392+0000","last_fullsized":"2023-12-13T23:57:19.530392+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:47:56.294820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861939+0000","last_change":"2023-12-13T23:57:17.770737+0000","last_active":"2023-12-13T23:57:19.861939+0000","last_peered":"2023-12-13T23:57:19.861939+0000","last_clean":"2023-12-13T23:57:19.861939+0000","last_became_active":"2023-12-13T23:57:17.770476+0000","last_became_peered":"2023-12-13T23:57:17.770476+0000","last_unstale":"2023-12-13T23:57:19.861939+0000","last_undegraded":"2023-12-13T23:57:19.861939+0000","last_fullsized":"2023-12-13T23:57:19.861939+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:36:22.300599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761479+0000","last_change":"2023-12-13T23:57:16.997540+0000","last_active":"2023-12-13T23:57:18.761479+0000","last_peered":"2023-12-13T23:57:18.761479+0000","last_clean":"2023-12-13T23:57:18.761479+0000","last_became_active":"2023-12-13T23:57:16.997015+0000","last_became_peered":"2023-12-13T23:57:16.997015+0000","last_unstale":"2023-12-13T23:57:18.761479+0000","last_undegraded":"2023-12-13T23:57:18.761479+0000","last_fullsized":"2023-12-13T23:57:18.761479+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337456+0000","last_change":"2023-12-13T23:57:17.773390+0000","last_active":"2023-12-13T23:57:19.337456+0000","last_peered":"2023-12-13T23:57:19.337456+0000","last_clean":"2023-12-13T23:57:19.337456+0000","last_became_active":"2023-12-13T23:57:17.773148+0000","last_became_peered":"2023-12-13T23:57:17.773148+0000","last_unstale":"2023-12-13T23:57:19.337456+0000","last_undegraded":"2023-12-13T23:57:19.337456+0000","last_fullsized":"2023-12-13T23:57:19.337456+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:49:50.405914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"49'8","reported_seq":20,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861702+0000","last_change":"2023-12-13T23:57:16.999816+0000","last_active":"2023-12-13T23:57:19.861702+0000","last_peered":"2023-12-13T23:57:19.861702+0000","last_clean":"2023-12-13T23:57:19.861702+0000","last_became_active":"2023-12-13T23:57:16.999571+0000","last_became_peered":"2023-12-13T23:57:16.999571+0000","last_unstale":"2023-12-13T23:57:19.861702+0000","last_undegraded":"2023-12-13T23:57:19.861702+0000","last_fullsized":"2023-12-13T23:57:19.861702+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:07:00.782128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861710+0000","last_change":"2023-12-13T23:57:17.770329+0000","last_active":"2023-12-13T23:57:19.861710+0000","last_peered":"2023-12-13T23:57:19.861710+0000","last_clean":"2023-12-13T23:57:19.861710+0000","last_became_active":"2023-12-13T23:57:17.770052+0000","last_became_peered":"2023-12-13T23:57:17.770052+0000","last_unstale":"2023-12-13T23:57:19.861710+0000","last_undegraded":"2023-12-13T23:57:19.861710+0000","last_fullsized":"2023-12-13T23:57:19.861710+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:47:48.928358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337580+0000","last_change":"2023-12-13T23:57:16.997082+0000","last_active":"2023-12-13T23:57:19.337580+0000","last_peered":"2023-12-13T23:57:19.337580+0000","last_clean":"2023-12-13T23:57:19.337580+0000","last_became_active":"2023-12-13T23:57:16.996807+0000","last_became_peered":"2023-12-13T23:57:16.996807+0000","last_unstale":"2023-12-13T23:57:19.337580+0000","last_undegraded":"2023-12-13T23:57:19.337580+0000","last_fullsized":"2023-12-13T23:57:19.337580+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861413+0000","last_change":"2023-12-13T23:57:17.773144+0000","last_active":"2023-12-13T23:57:19.861413+0000","last_peered":"2023-12-13T23:57:19.861413+0000","last_clean":"2023-12-13T23:57:19.861413+0000","last_became_active":"2023-12-13T23:57:17.772875+0000","last_became_peered":"2023-12-13T23:57:17.772875+0000","last_unstale":"2023-12-13T23:57:19.861413+0000","last_undegraded":"2023-12-13T23:57:19.861413+0000","last_fullsized":"2023-12-13T23:57:19.861413+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:28:03.359766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760028+0000","last_change":"2023-12-13T23:57:16.997740+0000","last_active":"2023-12-13T23:57:18.760028+0000","last_peered":"2023-12-13T23:57:18.760028+0000","last_clean":"2023-12-13T23:57:18.760028+0000","last_became_active":"2023-12-13T23:57:16.997479+0000","last_became_peered":"2023-12-13T23:57:16.997479+0000","last_unstale":"2023-12-13T23:57:18.760028+0000","last_undegraded":"2023-12-13T23:57:18.760028+0000","last_fullsized":"2023-12-13T23:57:18.760028+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:55:16.107027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.761454+0000","last_change":"2023-12-13T23:57:17.767817+0000","last_active":"2023-12-13T23:57:18.761454+0000","last_peered":"2023-12-13T23:57:18.761454+0000","last_clean":"2023-12-13T23:57:18.761454+0000","last_became_active":"2023-12-13T23:57:17.767439+0000","last_became_peered":"2023-12-13T23:57:17.767439+0000","last_unstale":"2023-12-13T23:57:18.761454+0000","last_undegraded":"2023-12-13T23:57:18.761454+0000","last_fullsized":"2023-12-13T23:57:18.761454+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T05:27:41.145805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"49'1","reported_seq":13,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530614+0000","last_change":"2023-12-13T23:57:16.996007+0000","last_active":"2023-12-13T23:57:19.530614+0000","last_peered":"2023-12-13T23:57:19.530614+0000","last_clean":"2023-12-13T23:57:19.530614+0000","last_became_active":"2023-12-13T23:57:16.995673+0000","last_became_peered":"2023-12-13T23:57:16.995673+0000","last_unstale":"2023-12-13T23:57:19.530614+0000","last_undegraded":"2023-12-13T23:57:19.530614+0000","last_fullsized":"2023-12-13T23:57:19.530614+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:21:23.704351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861525+0000","last_change":"2023-12-13T23:57:17.769082+0000","last_active":"2023-12-13T23:57:19.861525+0000","last_peered":"2023-12-13T23:57:19.861525+0000","last_clean":"2023-12-13T23:57:19.861525+0000","last_became_active":"2023-12-13T23:57:17.768807+0000","last_became_peered":"2023-12-13T23:57:17.768807+0000","last_unstale":"2023-12-13T23:57:19.861525+0000","last_undegraded":"2023-12-13T23:57:19.861525+0000","last_fullsized":"2023-12-13T23:57:19.861525+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:32.100913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530621+0000","last_change":"2023-12-13T23:57:16.997529+0000","last_active":"2023-12-13T23:57:19.530621+0000","last_peered":"2023-12-13T23:57:19.530621+0000","last_clean":"2023-12-13T23:57:19.530621+0000","last_became_active":"2023-12-13T23:57:16.997242+0000","last_became_peered":"2023-12-13T23:57:16.997242+0000","last_unstale":"2023-12-13T23:57:19.530621+0000","last_undegraded":"2023-12-13T23:57:19.530621+0000","last_fullsized":"2023-12-13T23:57:19.530621+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:35:56.043280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141646+0000","last_change":"2023-12-13T23:57:17.772911+0000","last_active":"2023-12-13T23:57:19.141646+0000","last_peered":"2023-12-13T23:57:19.141646+0000","last_clean":"2023-12-13T23:57:19.141646+0000","last_became_active":"2023-12-13T23:57:17.769997+0000","last_became_peered":"2023-12-13T23:57:17.769997+0000","last_unstale":"2023-12-13T23:57:19.141646+0000","last_undegraded":"2023-12-13T23:57:19.141646+0000","last_fullsized":"2023-12-13T23:57:19.141646+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:57:36.107177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861571+0000","last_change":"2023-12-13T23:57:16.998017+0000","last_active":"2023-12-13T23:57:19.861571+0000","last_peered":"2023-12-13T23:57:19.861571+0000","last_clean":"2023-12-13T23:57:19.861571+0000","last_became_active":"2023-12-13T23:57:16.997742+0000","last_became_peered":"2023-12-13T23:57:16.997742+0000","last_unstale":"2023-12-13T23:57:19.861571+0000","last_undegraded":"2023-12-13T23:57:19.861571+0000","last_fullsized":"2023-12-13T23:57:19.861571+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:45:49.347694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530579+0000","last_change":"2023-12-13T23:57:17.769969+0000","last_active":"2023-12-13T23:57:19.530579+0000","last_peered":"2023-12-13T23:57:19.530579+0000","last_clean":"2023-12-13T23:57:19.530579+0000","last_became_active":"2023-12-13T23:57:17.769688+0000","last_became_peered":"2023-12-13T23:57:17.769688+0000","last_unstale":"2023-12-13T23:57:19.530579+0000","last_undegraded":"2023-12-13T23:57:19.530579+0000","last_fullsized":"2023-12-13T23:57:19.530579+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T07:53:06.342217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861543+0000","last_change":"2023-12-13T23:57:17.003382+0000","last_active":"2023-12-13T23:57:19.861543+0000","last_peered":"2023-12-13T23:57:19.861543+0000","last_clean":"2023-12-13T23:57:19.861543+0000","last_became_active":"2023-12-13T23:57:17.002947+0000","last_became_peered":"2023-12-13T23:57:17.002947+0000","last_unstale":"2023-12-13T23:57:19.861543+0000","last_undegraded":"2023-12-13T23:57:19.861543+0000","last_fullsized":"2023-12-13T23:57:19.861543+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:49:49.404005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337425+0000","last_change":"2023-12-13T23:57:17.769120+0000","last_active":"2023-12-13T23:57:19.337425+0000","last_peered":"2023-12-13T23:57:19.337425+0000","last_clean":"2023-12-13T23:57:19.337425+0000","last_became_active":"2023-12-13T23:57:17.768875+0000","last_became_peered":"2023-12-13T23:57:17.768875+0000","last_unstale":"2023-12-13T23:57:19.337425+0000","last_undegraded":"2023-12-13T23:57:19.337425+0000","last_fullsized":"2023-12-13T23:57:19.337425+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:32:55.124768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337157+0000","last_change":"2023-12-13T23:57:16.998048+0000","last_active":"2023-12-13T23:57:19.337157+0000","last_peered":"2023-12-13T23:57:19.337157+0000","last_clean":"2023-12-13T23:57:19.337157+0000","last_became_active":"2023-12-13T23:57:16.997558+0000","last_became_peered":"2023-12-13T23:57:16.997558+0000","last_unstale":"2023-12-13T23:57:19.337157+0000","last_undegraded":"2023-12-13T23:57:19.337157+0000","last_fullsized":"2023-12-13T23:57:19.337157+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:45:45.161603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861864+0000","last_change":"2023-12-13T23:57:17.770499+0000","last_active":"2023-12-13T23:57:19.861864+0000","last_peered":"2023-12-13T23:57:19.861864+0000","last_clean":"2023-12-13T23:57:19.861864+0000","last_became_active":"2023-12-13T23:57:17.770243+0000","last_became_peered":"2023-12-13T23:57:17.770243+0000","last_unstale":"2023-12-13T23:57:19.861864+0000","last_undegraded":"2023-12-13T23:57:19.861864+0000","last_fullsized":"2023-12-13T23:57:19.861864+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:17:07.816899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.759768+0000","last_change":"2023-12-13T23:57:16.994643+0000","last_active":"2023-12-13T23:57:18.759768+0000","last_peered":"2023-12-13T23:57:18.759768+0000","last_clean":"2023-12-13T23:57:18.759768+0000","last_became_active":"2023-12-13T23:57:16.993858+0000","last_became_peered":"2023-12-13T23:57:16.993858+0000","last_unstale":"2023-12-13T23:57:18.759768+0000","last_undegraded":"2023-12-13T23:57:18.759768+0000","last_fullsized":"2023-12-13T23:57:18.759768+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:36:32.635221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141463+0000","last_change":"2023-12-13T23:57:17.773937+0000","last_active":"2023-12-13T23:57:19.141463+0000","last_peered":"2023-12-13T23:57:19.141463+0000","last_clean":"2023-12-13T23:57:19.141463+0000","last_became_active":"2023-12-13T23:57:17.773710+0000","last_became_peered":"2023-12-13T23:57:17.773710+0000","last_unstale":"2023-12-13T23:57:19.141463+0000","last_undegraded":"2023-12-13T23:57:19.141463+0000","last_fullsized":"2023-12-13T23:57:19.141463+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:32:42.626381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"49'2","reported_seq":14,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530714+0000","last_change":"2023-12-13T23:57:17.002360+0000","last_active":"2023-12-13T23:57:19.530714+0000","last_peered":"2023-12-13T23:57:19.530714+0000","last_clean":"2023-12-13T23:57:19.530714+0000","last_became_active":"2023-12-13T23:57:17.001948+0000","last_became_peered":"2023-12-13T23:57:17.001948+0000","last_unstale":"2023-12-13T23:57:19.530714+0000","last_undegraded":"2023-12-13T23:57:19.530714+0000","last_fullsized":"2023-12-13T23:57:19.530714+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:37:47.093891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861853+0000","last_change":"2023-12-13T23:57:17.768960+0000","last_active":"2023-12-13T23:57:19.861853+0000","last_peered":"2023-12-13T23:57:19.861853+0000","last_clean":"2023-12-13T23:57:19.861853+0000","last_became_active":"2023-12-13T23:57:17.768692+0000","last_became_peered":"2023-12-13T23:57:17.768692+0000","last_unstale":"2023-12-13T23:57:19.861853+0000","last_undegraded":"2023-12-13T23:57:19.861853+0000","last_fullsized":"2023-12-13T23:57:19.861853+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:38:07.225635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.762302+0000","last_change":"2023-12-13T23:57:16.999504+0000","last_active":"2023-12-13T23:57:18.762302+0000","last_peered":"2023-12-13T23:57:18.762302+0000","last_clean":"2023-12-13T23:57:18.762302+0000","last_became_active":"2023-12-13T23:57:16.999123+0000","last_became_peered":"2023-12-13T23:57:16.999123+0000","last_unstale":"2023-12-13T23:57:18.762302+0000","last_undegraded":"2023-12-13T23:57:18.762302+0000","last_fullsized":"2023-12-13T23:57:18.762302+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:57:44.774717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861717+0000","last_change":"2023-12-13T23:57:17.770897+0000","last_active":"2023-12-13T23:57:19.861717+0000","last_peered":"2023-12-13T23:57:19.861717+0000","last_clean":"2023-12-13T23:57:19.861717+0000","last_became_active":"2023-12-13T23:57:17.770120+0000","last_became_peered":"2023-12-13T23:57:17.770120+0000","last_unstale":"2023-12-13T23:57:19.861717+0000","last_undegraded":"2023-12-13T23:57:19.861717+0000","last_fullsized":"2023-12-13T23:57:19.861717+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:42:24.215630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337219+0000","last_change":"2023-12-13T23:57:17.001998+0000","last_active":"2023-12-13T23:57:19.337219+0000","last_peered":"2023-12-13T23:57:19.337219+0000","last_clean":"2023-12-13T23:57:19.337219+0000","last_became_active":"2023-12-13T23:57:17.001741+0000","last_became_peered":"2023-12-13T23:57:17.001741+0000","last_unstale":"2023-12-13T23:57:19.337219+0000","last_undegraded":"2023-12-13T23:57:19.337219+0000","last_fullsized":"2023-12-13T23:57:19.337219+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:26:34.277360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337245+0000","last_change":"2023-12-13T23:57:17.773847+0000","last_active":"2023-12-13T23:57:19.337245+0000","last_peered":"2023-12-13T23:57:19.337245+0000","last_clean":"2023-12-13T23:57:19.337245+0000","last_became_active":"2023-12-13T23:57:17.772054+0000","last_became_peered":"2023-12-13T23:57:17.772054+0000","last_unstale":"2023-12-13T23:57:19.337245+0000","last_undegraded":"2023-12-13T23:57:19.337245+0000","last_fullsized":"2023-12-13T23:57:19.337245+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:02:14.954573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861737+0000","last_change":"2023-12-13T23:57:17.000763+0000","last_active":"2023-12-13T23:57:19.861737+0000","last_peered":"2023-12-13T23:57:19.861737+0000","last_clean":"2023-12-13T23:57:19.861737+0000","last_became_active":"2023-12-13T23:57:17.000524+0000","last_became_peered":"2023-12-13T23:57:17.000524+0000","last_unstale":"2023-12-13T23:57:19.861737+0000","last_undegraded":"2023-12-13T23:57:19.861737+0000","last_fullsized":"2023-12-13T23:57:19.861737+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T09:08:41.849881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141652+0000","last_change":"2023-12-13T23:57:17.775318+0000","last_active":"2023-12-13T23:57:19.141652+0000","last_peered":"2023-12-13T23:57:19.141652+0000","last_clean":"2023-12-13T23:57:19.141652+0000","last_became_active":"2023-12-13T23:57:17.775003+0000","last_became_peered":"2023-12-13T23:57:17.775003+0000","last_unstale":"2023-12-13T23:57:19.141652+0000","last_undegraded":"2023-12-13T23:57:19.141652+0000","last_fullsized":"2023-12-13T23:57:19.141652+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:26:27.497532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.861594+0000","last_change":"2023-12-13T23:57:17.003199+0000","last_active":"2023-12-13T23:57:19.861594+0000","last_peered":"2023-12-13T23:57:19.861594+0000","last_clean":"2023-12-13T23:57:19.861594+0000","last_became_active":"2023-12-13T23:57:17.002578+0000","last_became_peered":"2023-12-13T23:57:17.002578+0000","last_unstale":"2023-12-13T23:57:19.861594+0000","last_undegraded":"2023-12-13T23:57:19.861594+0000","last_fullsized":"2023-12-13T23:57:19.861594+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T00:21:34.393111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337352+0000","last_change":"2023-12-13T23:57:17.771702+0000","last_active":"2023-12-13T23:57:19.337352+0000","last_peered":"2023-12-13T23:57:19.337352+0000","last_clean":"2023-12-13T23:57:19.337352+0000","last_became_active":"2023-12-13T23:57:17.771453+0000","last_became_peered":"2023-12-13T23:57:17.771453+0000","last_unstale":"2023-12-13T23:57:19.337352+0000","last_undegraded":"2023-12-13T23:57:19.337352+0000","last_fullsized":"2023-12-13T23:57:19.337352+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:09:58.638616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:18.760992+0000","last_change":"2023-12-13T23:57:17.005167+0000","last_active":"2023-12-13T23:57:18.760992+0000","last_peered":"2023-12-13T23:57:18.760992+0000","last_clean":"2023-12-13T23:57:18.760992+0000","last_became_active":"2023-12-13T23:57:17.004864+0000","last_became_peered":"2023-12-13T23:57:17.004864+0000","last_unstale":"2023-12-13T23:57:18.760992+0000","last_undegraded":"2023-12-13T23:57:18.760992+0000","last_fullsized":"2023-12-13T23:57:18.760992+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T03:09:43.322823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.862229+0000","last_change":"2023-12-13T23:57:17.772589+0000","last_active":"2023-12-13T23:57:19.862229+0000","last_peered":"2023-12-13T23:57:19.862229+0000","last_clean":"2023-12-13T23:57:19.862229+0000","last_became_active":"2023-12-13T23:57:17.772317+0000","last_became_peered":"2023-12-13T23:57:17.772317+0000","last_unstale":"2023-12-13T23:57:19.862229+0000","last_undegraded":"2023-12-13T23:57:19.862229+0000","last_fullsized":"2023-12-13T23:57:19.862229+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:08:22.730122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.141514+0000","last_change":"2023-12-13T23:57:17.001151+0000","last_active":"2023-12-13T23:57:19.141514+0000","last_peered":"2023-12-13T23:57:19.141514+0000","last_clean":"2023-12-13T23:57:19.141514+0000","last_became_active":"2023-12-13T23:57:17.000823+0000","last_became_peered":"2023-12-13T23:57:17.000823+0000","last_unstale":"2023-12-13T23:57:19.141514+0000","last_undegraded":"2023-12-13T23:57:19.141514+0000","last_fullsized":"2023-12-13T23:57:19.141514+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T02:26:34.277360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":11,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.530358+0000","last_change":"2023-12-13T23:57:17.768951+0000","last_active":"2023-12-13T23:57:19.530358+0000","last_peered":"2023-12-13T23:57:19.530358+0000","last_clean":"2023-12-13T23:57:19.530358+0000","last_became_active":"2023-12-13T23:57:17.768649+0000","last_became_peered":"2023-12-13T23:57:17.768649+0000","last_unstale":"2023-12-13T23:57:19.530358+0000","last_undegraded":"2023-12-13T23:57:19.530358+0000","last_fullsized":"2023-12-13T23:57:19.530358+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":48,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:16.977342+0000","last_clean_scrub_stamp":"2023-12-13T23:57:16.977342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T11:49:00.328074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":12,"reported_epoch":50,"state":"active+clean","last_fresh":"2023-12-13T23:57:19.337451+0000","last_change":"2023-12-13T23:57:17.000835+0000","last_active":"2023-12-13T23:57:19.337451+0000","last_peered":"2023-12-13T23:57:19.337451+0000","last_clean":"2023-12-13T23:57:19.337451+0000","last_became_active":"2023-12-13T23:57:16.997260+0000","last_became_peered":"2023-12-13T23:57:16.997260+0000","last_unstale":"2023-12-13T23:57:19.337451+0000","last_undegraded":"2023-12-13T23:57:19.337451+0000","last_fullsized":"2023-12-13T23:57:19.337451+0000","mapping_epoch":47,"log_start":"0'0","ondisk_log_start":"0'0","created":47,"last_epoch_clean":48,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:57:15.975433+0000","last_clean_scrub_stamp":"2023-12-13T23:57:15.975433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T06:35:31.099004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":195,"reported_epoch":49,"state":"active+clean","last_fresh":"2023-12-13T23:57:17.758499+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:17.758499+0000","last_peered":"2023-12-13T23:57:17.758499+0000","last_clean":"2023-12-13T23:57:17.758499+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:17.758499+0000","last_undegraded":"2023-12-13T23:57:17.758499+0000","last_fullsized":"2023-12-13T23:57:17.758499+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574880,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1196032,"data_stored":837070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.51200000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.496},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60499999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.38}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.40400000000000003}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.56299999999999994}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":602112,"data_stored":247230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.52400000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.57399999999999995}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.624}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.60399999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.35399999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.376},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.496}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.432}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27600,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344240,"statfs":{"total":96636764160,"available":96608501760,"internally_reserved":0,"allocated":593920,"data_stored":246578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.432},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.45100000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.502}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58899999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.66500000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.71399999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574881,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":598016,"data_stored":247160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.44700000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58399999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.49099999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.46300000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.52400000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.311},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41999999999999998}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.374}]}]},{"osd":3,"up_from":13,"seq":55834574880,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27616,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344224,"statfs":{"total":96636764160,"available":96608485376,"internally_reserved":0,"allocated":610304,"data_stored":248212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.25},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.37}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.32200000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.39000000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46200000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.496},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.51000000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27612,"kb_used_data":592,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344228,"statfs":{"total":96636764160,"available":96608489472,"internally_reserved":0,"allocated":606208,"data_stored":248190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.30399999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.36399999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.379}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.63900000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.495},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71699999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55500000000000005}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.67200000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27604,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":589824,"data_stored":240438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.437},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.40100000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.50600000000000001}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.64900000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.63800000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1183744,"data_stored":830702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.30199999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.376}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.31900000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.53100000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.505},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.56000000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.40799999999999997}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:21.276 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-12-13T23:57:21.276 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-12-13T23:57:21.418 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.419+0000 7f8d3da09640 1 Processor -- start 2023-12-13T23:57:21.419 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.419+0000 7f8d3da09640 1 -- start start 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d3da09640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 0x7f8d380b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d3da09640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8d380d3850 0x7f8d380cf9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d3da09640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d380d00f0 con 0x7f8d380b43a0 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d3da09640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d380cff00 con 0x7f8d380d3850 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d3da09640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380d1420 con 0x7f8d3815d420 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d377fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8d380d3850 0x7f8d380cf9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d367fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8d380b43a0 0x7f8d380d3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59194/0 (socket says 172.21.15.124:59194) 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d367fc640 1 -- 172.21.15.124:0/2830507211 learned_addr learned my addr 172.21.15.124:0/2830507211 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d36ffd640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 0x7f8d380b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d377fe640 1 -- 172.21.15.124:0/2830507211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 msgr2=0x7f8d380b2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.420+0000 7f8d377fe640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 0x7f8d380b2b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d377fe640 1 -- 172.21.15.124:0/2830507211 >> v1:172.21.15.124:6789/0 conn(0x7f8d380b43a0 legacy=0x7f8d380d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.420 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d377fe640 1 -- 172.21.15.124:0/2830507211 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380d03d0 con 0x7f8d380d3850 2023-12-13T23:57:21.421 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d36ffd640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 0x7f8d380b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d377fe640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380d3850 0x7f8d380cf9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097d0 tx=0x7f8d3002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d11c27a3dc5eb6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d35ffb640 1 -- 172.21.15.124:0/2830507211 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30031800 con 0x7f8d380d3850 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d35ffb640 1 -- 172.21.15.124:0/2830507211 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d30031960 con 0x7f8d380d3850 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.421+0000 7f8d35ffb640 1 -- 172.21.15.124:0/2830507211 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d30031cf0 con 0x7f8d380d3850 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 -- 172.21.15.124:0/2830507211 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380d3850 msgr2=0x7f8d380cf9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380d3850 0x7f8d380cf9c0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f8d300097d0 tx=0x7f8d3002b8a0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 -- 172.21.15.124:0/2830507211 shutdown_connections 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380d3850 0x7f8d380cf9c0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 --2- 172.21.15.124:0/2830507211 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d3815d420 0x7f8d380b2b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 -- 172.21.15.124:0/2830507211 >> 172.21.15.124:0/2830507211 conn(0x7f8d3808b000 msgr2=0x7f8d3809c580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.422 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 -- 172.21.15.124:0/2830507211 shutdown_connections 2023-12-13T23:57:21.423 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.422+0000 7f8d3da09640 1 -- 172.21.15.124:0/2830507211 wait complete. 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.423+0000 7f8d3da09640 1 Processor -- start 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.423+0000 7f8d3da09640 1 -- start start 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d380d3850 0x7f8d381f6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3815d420 0x7f8d381f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d381f7140 con 0x7f8d380b43a0 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380b10b0 con 0x7f8d3815d420 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d3da09640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d380b8de0 con 0x7f8d380d3850 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57116/0 (socket says 172.21.15.124:57116) 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 -- 172.21.15.124:0/1260501351 learned_addr learned my addr 172.21.15.124:0/1260501351 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:21.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d377fe640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3815d420 0x7f8d381f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d367fc640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d380d3850 0x7f8d381f6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 -- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d380d3850 msgr2=0x7f8d381f6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d380d3850 0x7f8d381f6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 -- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3815d420 msgr2=0x7f8d381f2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3815d420 0x7f8d381f2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.424+0000 7f8d36ffd640 1 -- 172.21.15.124:0/1260501351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30009400 con 0x7f8d380b43a0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d36ffd640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800ce00 tx=0x7f8d2800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2800f890 con 0x7f8d380b43a0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2800f9f0 con 0x7f8d380b43a0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d380b1370 con 0x7f8d380b43a0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2800fdc0 con 0x7f8d380b43a0 2023-12-13T23:57:21.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.425+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d380b8a70 con 0x7f8d380b43a0 2023-12-13T23:57:21.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.426+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d38069400 con 0x7f8d380b43a0 2023-12-13T23:57:21.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.426+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8d2800b040 con 0x7f8d380b43a0 2023-12-13T23:57:21.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.427+0000 7f8d1ffff640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 0x7f8d2406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.427 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.427+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d2808a320 con 0x7f8d380b43a0 2023-12-13T23:57:21.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.427+0000 7f8d367fc640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 0x7f8d2406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.428+0000 7f8d367fc640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 0x7f8d2406d680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8d380a10e0 tx=0x7f8d2c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.431+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8d280550e0 con 0x7f8d380b43a0 2023-12-13T23:57:21.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.662+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8d380d3fe0 con 0x7f8d380b43a0 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.664+0000 7f8d1ffff640 1 -- 172.21.15.124:0/1260501351 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8d2801d0b0 con 0x7f8d380b43a0 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:57:21.663 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-12-13T23:57:21.664 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 msgr2=0x7f8d2406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 0x7f8d2406d680 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8d380a10e0 tx=0x7f8d2c009040 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 msgr2=0x7f8d3815e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f8d2800ce00 tx=0x7f8d2800a650 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 shutdown_connections 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d2406b1c0 0x7f8d2406d680 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d3815d420 0x7f8d381f2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d380d3850 0x7f8d381f6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 --2- 172.21.15.124:0/1260501351 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d380b43a0 0x7f8d3815e230 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 >> 172.21.15.124:0/1260501351 conn(0x7f8d3808b000 msgr2=0x7f8d3806cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 shutdown_connections 2023-12-13T23:57:21.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.666+0000 7f8d3da09640 1 -- 172.21.15.124:0/1260501351 wait complete. 2023-12-13T23:57:21.676 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:21.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.776+0000 7f1f21a9b640 1 Processor -- start 2023-12-13T23:57:21.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.776+0000 7f1f21a9b640 1 -- start start 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1f1c15d330 0x7f1f1c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1f1c0d2910 0x7f1f1c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c0b2b10 con 0x7f1f1c15d330 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f1c0d3b90 con 0x7f1f1c0d2910 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f21a9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0d1590 con 0x7f1f1c0d3730 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f1b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1f1c15d330 0x7f1f1c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f1b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1f1c15d330 0x7f1f1c0b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57118/0 (socket says 172.21.15.124:57118) 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.777+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/2494185121 learned_addr learned my addr 172.21.15.124:0/2494185121 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1bfff640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/2494185121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 msgr2=0x7f1f1c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/2494185121 >> v2:172.21.15.203:3300/0 conn(0x7f1f1c0d2910 msgr2=0x7f1f1c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f12ffd640 1 --2- 172.21.15.124:0/2494185121 >> v2:172.21.15.203:3300/0 conn(0x7f1f1c0d2910 0x7f1f1c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/2494185121 >> v2:172.21.15.203:3300/0 conn(0x7f1f1c0d2910 0x7f1f1c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/2494185121 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0d1870 con 0x7f1f1c15d330 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.778+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0b8b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0d1540 tx=0x7f1f0c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7943d106c192c5a8 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f1affd640 1 -- 172.21.15.124:0/2494185121 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c0317e0 con 0x7f1f1c15d330 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f1affd640 1 -- 172.21.15.124:0/2494185121 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c031940 con 0x7f1f1c15d330 2023-12-13T23:57:21.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f1affd640 1 -- 172.21.15.124:0/2494185121 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c031cd0 con 0x7f1f1c15d330 2023-12-13T23:57:21.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 -- 172.21.15.124:0/2494185121 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c15d330 msgr2=0x7f1f1c0b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0b8b80 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0d1540 tx=0x7f1f0c02b880 comp rx=0 tx=0).stop 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 -- 172.21.15.124:0/2494185121 shutdown_connections 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/2494185121 >> v2:172.21.15.203:3300/0 conn(0x7f1f1c0d2910 0x7f1f1c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/2494185121 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0b8b80 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 -- 172.21.15.124:0/2494185121 >> 172.21.15.124:0/2494185121 conn(0x7f1f1c072020 msgr2=0x7f1f1c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 -- 172.21.15.124:0/2494185121 shutdown_connections 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.779+0000 7f1f21a9b640 1 -- 172.21.15.124:0/2494185121 wait complete. 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.780+0000 7f1f21a9b640 1 Processor -- start 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 -- start start 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c15b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0ca800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c0c7bf0 con 0x7f1f1c0d2910 2023-12-13T23:57:21.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1f3200 con 0x7f1f1c15d330 2023-12-13T23:57:21.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f21a9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1c1f34e0 con 0x7f1f1c0d3730 2023-12-13T23:57:21.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f1b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f1b7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57126/0 (socket says 172.21.15.124:57126) 2023-12-13T23:57:21.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/3530877147 learned_addr learned my addr 172.21.15.124:0/3530877147 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:21.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f1bfff640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0ca800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.781+0000 7f1f12ffd640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c15b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 msgr2=0x7f1f1c15b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c15b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c15d330 msgr2=0x7f1f1c0ca800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0ca800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 -- 172.21.15.124:0/3530877147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c0094d0 con 0x7f1f1c0d2910 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f1b7fe640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c032900 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c032a60 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1c1f37c0 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f0c032e10 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.782+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1c0cad40 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.783+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f1c069400 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.783+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1f0c04e450 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.784+0000 7f1f18ff9640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 0x7f1efc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.784+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1f0c0abd20 con 0x7f1f1c0d2910 2023-12-13T23:57:21.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.784+0000 7f1f12ffd640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 0x7f1efc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:21.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.784+0000 7f1f12ffd640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 0x7f1efc06d680 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0c9490 tx=0x7f1f04008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:21.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.788+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1f0c076b00 con 0x7f1f1c0d2910 2023-12-13T23:57:21.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.989+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f1c0ca5e0 con 0x7f1f1c0d2910 2023-12-13T23:57:21.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.990+0000 7f1f18ff9640 1 -- 172.21.15.124:0/3530877147 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f1f0c07a120 con 0x7f1f1c0d2910 2023-12-13T23:57:21.989 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:21.989 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:21.990 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 msgr2=0x7f1efc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 0x7f1efc06d680 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c0c9490 tx=0x7f1f04008040 comp rx=0 tx=0).stop 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 msgr2=0x7f1f1c1f2c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c000c00 tx=0x7f1f0c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 shutdown_connections 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1efc06b1c0 0x7f1efc06d680 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1f1c15d330 0x7f1f1c0ca800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1f1c0d3730 0x7f1f1c15b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 --2- 172.21.15.124:0/3530877147 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f1c0d2910 0x7f1f1c1f2c20 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:21.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 >> 172.21.15.124:0/3530877147 conn(0x7f1f1c072020 msgr2=0x7f1f1c0b8730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:21.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 shutdown_connections 2023-12-13T23:57:21.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:21.992+0000 7f1f21a9b640 1 -- 172.21.15.124:0/3530877147 wait complete. 2023-12-13T23:57:22.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 22, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-12-13T23:57:17.326453+0000', 'modified': '2023-12-13T23:57:19.327808+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5910}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5910': {'gid': 5910, 'name': 'b', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.203:6837/2421641365', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.203:6835', 'nonce': 2421641365}, {'type': 'v1', 'addr': '172.21.15.203:6837', 'nonce': 2421641365}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701272860131327, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-12-13T23:57:22.405 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-12-13T23:57:22.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-12-13T23:57:22.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.508+0000 7fc9086a6640 1 Processor -- start 2023-12-13T23:57:22.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.508+0000 7fc9086a6640 1 -- start start 2023-12-13T23:57:22.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.509+0000 7fc9086a6640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc900143490 0x7fc9000b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.509+0000 7fc9086a6640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 0x7fc9000b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.509+0000 7fc9086a6640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90015e2a0 con 0x7fc90015e730 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.509+0000 7fc9086a6640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90015b520 con 0x7fc900143490 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.509+0000 7fc9086a6640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900143870 con 0x7fc90015abf0 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc900143490 0x7fc9000b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc905c41640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc90015e730 0x7fc90015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59202/0 (socket says 172.21.15.124:59202) 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc900143490 0x7fc9000b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57142/0 (socket says 172.21.15.124:57142) 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc905c41640 1 -- 172.21.15.124:0/3986322224 learned_addr learned my addr 172.21.15.124:0/3986322224 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906c43640 1 --2- 172.21.15.124:0/3986322224 >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 0x7fc9000b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 -- 172.21.15.124:0/3986322224 >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 msgr2=0x7fc9000b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 --2- 172.21.15.124:0/3986322224 >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 0x7fc9000b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 -- 172.21.15.124:0/3986322224 >> v1:172.21.15.124:6789/0 conn(0x7fc90015e730 legacy=0x7fc90015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906442640 1 -- 172.21.15.124:0/3986322224 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90015b240 con 0x7fc900143490 2023-12-13T23:57:22.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc905440640 1 -- 172.21.15.124:0/3986322224 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611854364 0 0) 0x7fc90015e2a0 con 0x7fc90015e730 2023-12-13T23:57:22.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.510+0000 7fc906c43640 1 --2- 172.21.15.124:0/3986322224 >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 0x7fc9000b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:22.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc906442640 1 --2- 172.21.15.124:0/3986322224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc900143490 0x7fc9000b0cf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc9000b4ca0 tx=0x7fc8fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b10d011ad555e59b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc905440640 1 -- 172.21.15.124:0/3986322224 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc031830 con 0x7fc900143490 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc905440640 1 -- 172.21.15.124:0/3986322224 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc031990 con 0x7fc900143490 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc905440640 1 -- 172.21.15.124:0/3986322224 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc031d20 con 0x7fc900143490 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 -- 172.21.15.124:0/3986322224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc900143490 msgr2=0x7fc9000b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 --2- 172.21.15.124:0/3986322224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc900143490 0x7fc9000b0cf0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7fc9000b4ca0 tx=0x7fc8fc02b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 -- 172.21.15.124:0/3986322224 shutdown_connections 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 --2- 172.21.15.124:0/3986322224 >> v2:172.21.15.203:3300/0 conn(0x7fc90015abf0 0x7fc9000b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 --2- 172.21.15.124:0/3986322224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc900143490 0x7fc9000b0cf0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 -- 172.21.15.124:0/3986322224 >> 172.21.15.124:0/3986322224 conn(0x7fc90006a4d0 msgr2=0x7fc90006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 -- 172.21.15.124:0/3986322224 shutdown_connections 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.511+0000 7fc9086a6640 1 -- 172.21.15.124:0/3986322224 wait complete. 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.512+0000 7fc9086a6640 1 Processor -- start 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 -- start start 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc900143490 0x7fc9001f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc90015abf0 0x7fc90015c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000b0940 con 0x7fc90015e730 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900131b30 con 0x7fc900143490 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc9086a6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900131e10 con 0x7fc90015abf0 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57144/0 (socket says 172.21.15.124:57144) 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.513+0000 7fc906c43640 1 -- 172.21.15.124:0/2945225621 learned_addr learned my addr 172.21.15.124:0/2945225621 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 -- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc90015abf0 msgr2=0x7fc90015c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc90015abf0 0x7fc90015c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 -- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc900143490 msgr2=0x7fc9001f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906442640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc900143490 0x7fc9001f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc900143490 0x7fc9001f4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 -- 172.21.15.124:0/2945225621 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc009400 con 0x7fc90015e730 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc906c43640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cef0 tx=0x7fc8f800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc8f6ffd640 1 -- 172.21.15.124:0/2945225621 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800d840 con 0x7fc90015e730 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc8f6ffd640 1 -- 172.21.15.124:0/2945225621 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f800d9c0 con 0x7fc90015e730 2023-12-13T23:57:22.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9001320f0 con 0x7fc90015e730 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc8f6ffd640 1 -- 172.21.15.124:0/2945225621 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800dd70 con 0x7fc90015e730 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.514+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9001326b0 con 0x7fc90015e730 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.515+0000 7fc8f6ffd640 1 -- 172.21.15.124:0/2945225621 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc8f801e3b0 con 0x7fc90015e730 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.516+0000 7fc8f6ffd640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 0x7fc8e406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.516+0000 7fc8f6ffd640 1 -- 172.21.15.124:0/2945225621 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8f808a4c0 con 0x7fc90015e730 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.516+0000 7fc906442640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 0x7fc8e406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.516+0000 7fc906442640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 0x7fc8e406d680 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036000 tx=0x7fc8fc008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.517+0000 7fc90b23e740 1 Processor -- start 2023-12-13T23:57:22.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.517+0000 7fc90b23e740 1 -- start start 2023-12-13T23:57:22.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 0x562789741ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 0x562789771b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627896b5020 con 0x56278976ec70 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627896b2710 con 0x56278976e890 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627896ae000 con 0x562789742000 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57154/0 (socket says 172.21.15.124:57154) 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 -- 172.21.15.124:0/2986450526 learned_addr learned my addr 172.21.15.124:0/2986450526 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc906442640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 0x562789741ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc906c43640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 0x562789771b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 -- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 msgr2=0x562789771b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 0x562789771b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 -- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 msgr2=0x562789741ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 0x562789741ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 -- 172.21.15.124:0/2986450526 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562789670c90 con 0x56278976ec70 2023-12-13T23:57:22.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc906c43640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 0x562789771b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc906442640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 0x562789741ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc905c41640 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0001660 tx=0x7fc8f0001b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71580c7251d3ed1b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc8e3fff640 1 -- 172.21.15.124:0/2986450526 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f00074e0 con 0x56278976ec70 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc8e3fff640 1 -- 172.21.15.124:0/2986450526 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0001d70 con 0x56278976ec70 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.518+0000 7fc8e3fff640 1 -- 172.21.15.124:0/2986450526 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f00049e0 con 0x56278976ec70 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 -- 172.21.15.124:0/2986450526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 msgr2=0x562789771620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0001660 tx=0x7fc8f0001b20 comp rx=0 tx=0).stop 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 -- 172.21.15.124:0/2986450526 shutdown_connections 2023-12-13T23:57:22.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x562789742000 0x562789771b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976ec70 0x562789771620 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 --2- 172.21.15.124:0/2986450526 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976e890 0x562789741ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 -- 172.21.15.124:0/2986450526 >> 172.21.15.124:0/2986450526 conn(0x562789743d30 msgr2=0x56278978c620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 -- 172.21.15.124:0/2986450526 shutdown_connections 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.519+0000 7fc90b23e740 1 -- 172.21.15.124:0/2986450526 wait complete. 2023-12-13T23:57:22.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.520+0000 7fc90b23e740 1 -- start start 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.520+0000 7fc90b23e740 1 Processor -- start 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278978fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x562789667e00 con 0x56278979bee0 2023-12-13T23:57:22.534 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562789667aa0 con 0x56278976ec70 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x56278962d270 con 0x56278976e890 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 10 client.0 _ll_register_callbacks cb 0x5627893bfac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57158/0 (socket says 172.21.15.124:57158) 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 -- 172.21.15.124:0/3497218359 learned_addr learned my addr 172.21.15.124:0/3497218359 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.535 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906442640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278978fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc905c41640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 -- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 msgr2=0x56278978fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278978fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 -- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 msgr2=0x56278976d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 -- 172.21.15.124:0/3497218359 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f0001260 con 0x56278979bee0 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc906c43640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8088990 tx=0x7fc8f80590b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66fcc0061b54459e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc8e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:57:22.536 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8055680 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8f80557e0 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627896a6e90 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8f808a0c0 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 10 client.6042 fetch_fsmap learned FSMap version 22 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x5627896ac680 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.521+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5627896a6e90 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 22) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc8f8008040 con 0x56278979bee0 2023-12-13T23:57:22.537 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 10 client.6042 fetch_fsmap finished waiting for FSMap version 22 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 10 client.6042 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 20 client.6042 populate_metadata read hostname 'smithi124' 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x5627897a7640 0x5627897a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 4 client.6042 mds_command: new command op to 5910 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 --> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627896a6bb0 con 0x5627897a7640 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc906442640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x5627897a7640 0x5627897a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc906442640 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x5627897a7640 0x5627897a9a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43b797ca05bf8a73 server_cookie=a1cf5583deb3b5ad in_seq=0 out_seq=0 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.522+0000 7fc8e17fa640 10 client.6042 ms_handle_connect on v2:172.21.15.203:6835/2421641365 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc8e17fa640 1 -- 172.21.15.124:0/3497218359 <== mds.0 v2:172.21.15.203:6835/2421641365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5627896a6bb0 con 0x5627897a7640 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc8e17fa640 10 client.6042 handle_command_reply: tid=0 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 client.6042 shutdown 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 msgr2=0x562789784f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.538 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8088990 tx=0x7fc8f80590b0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 shutdown_connections 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x5627897a7640 0x5627897a9a30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x562789784f90 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 --2- 172.21.15.124:0/3497218359 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278978fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc8e17fa640 0 client.6042 ms_handle_reset on v2:172.21.15.203:6835/2421641365 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 >> 172.21.15.124:0/3497218359 conn(0x562789742000 msgr2=0x562789745710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 shutdown_connections 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 1 -- 172.21.15.124:0/3497218359 wait complete. 2023-12-13T23:57:22.539 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.523+0000 7fc90b23e740 20 client.6042 trim_cache size 0 max 16384 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.539+0000 7fc90b23e740 1 Processor -- start 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.539+0000 7fc90b23e740 1 -- start start 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.539+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278976d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.539+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc90b23e740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278976deb0 con 0x56278979bee0 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562789662e70 con 0x56278976ec70 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x56278976e680 con 0x56278976e890 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57174/0 (socket says 172.21.15.124:57174) 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 -- 172.21.15.124:0/1394453754 learned_addr learned my addr 172.21.15.124:0/1394453754 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906442640 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278976d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc905c41640 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 -- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 msgr2=0x56278976d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278976d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 -- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 msgr2=0x56278976d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 -- 172.21.15.124:0/1394453754 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562789649d00 con 0x56278979bee0 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc906c43640 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8058c20 tx=0x7fc8f808a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a0b6e2b20afe082 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.547 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc8f4ff9640 1 -- 172.21.15.124:0/1394453754 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f808a0c0 con 0x56278979bee0 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc8f4ff9640 1 -- 172.21.15.124:0/1394453754 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f807b070 con 0x56278979bee0 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.540+0000 7fc8f4ff9640 1 -- 172.21.15.124:0/1394453754 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8007a00 con 0x56278979bee0 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- 172.21.15.124:0/1394453754 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 msgr2=0x56278978e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8058c20 tx=0x7fc8f808a050 comp rx=0 tx=0).stop 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- 172.21.15.124:0/1394453754 shutdown_connections 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278976d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278979bee0 0x56278978e880 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1394453754 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278976e890 0x56278976d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- 172.21.15.124:0/1394453754 >> 172.21.15.124:0/1394453754 conn(0x562789742000 msgr2=0x562789787780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- 172.21.15.124:0/1394453754 shutdown_connections 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- 172.21.15.124:0/1394453754 wait complete. 2023-12-13T23:57:22.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 -- start start 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.541+0000 7fc90b23e740 1 Processor -- start 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278978d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278979bee0 0x5627897829c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x562789783830 con 0x56278976e890 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x562789747a80 con 0x56278976ec70 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x5627897fa3d0 con 0x56278979bee0 2023-12-13T23:57:22.549 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc90b23e740 10 client.0 _ll_register_callbacks cb 0x7fc8f0007c60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc906442640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc906442640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57180/0 (socket says 172.21.15.124:57180) 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc906442640 1 -- 172.21.15.124:0/1644081398 learned_addr learned my addr 172.21.15.124:0/1644081398 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc905c41640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278978d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.542+0000 7fc906c43640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278979bee0 0x5627897829c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 -- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278979bee0 msgr2=0x5627897829c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278979bee0 0x5627897829c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 -- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 msgr2=0x56278978d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278978d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 -- 172.21.15.124:0/1644081398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627896ac680 con 0x56278976e890 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc906442640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc031d80 tx=0x7fc8fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4e18ed45552c40 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.124:3300/0 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc040070 con 0x56278976e890 2023-12-13T23:57:22.550 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mon.0 v2:172.21.15.124:3300/0 2 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc8fc003ae0 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8f0001d70 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8fc003ca0 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 10 client.6048 fetch_fsmap learned FSMap version 22 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,fsmap=22,monmap=2+}) v3 -- 0x56278966f4f0 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mon.0 v2:172.21.15.124:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f0001d70 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mon.0 v2:172.21.15.124:3300/0 5 ==== fsmap(e 22) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fc8fc053040 con 0x56278976e890 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 10 client.6048 fetch_fsmap finished waiting for FSMap version 22 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 10 client.6048 resolve_mds: resolved name 'b' to daemon mds.b 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 20 client.6048 populate_metadata read hostname 'smithi124' 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x56278976d250 0x56278976d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 4 client.6048 mds_command: new command op to 5910 tid=0 multi_id=0 [{"prefix": "status"}] 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.543+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 --> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5627893ae6c0 con 0x56278976d250 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc905c41640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x56278976d250 0x56278976d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc905c41640 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x56278976d250 0x56278976d630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6f4b1f619d0228f server_cookie=a456baea62ba7b66 in_seq=0 out_seq=0 2023-12-13T23:57:22.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc8e17fa640 10 client.6048 ms_handle_connect on v2:172.21.15.203:6835/2421641365 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc8e17fa640 1 -- 172.21.15.124:0/1644081398 <== mds.0 v2:172.21.15.203:6835/2421641365 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5627893ae6c0 con 0x56278976d250 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc8e17fa640 10 client.6048 handle_command_reply: tid=0 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 client.6048 shutdown 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 msgr2=0x5627897f9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc031d80 tx=0x7fc8fc004730 comp rx=0 tx=0).stop 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 shutdown_connections 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:6835/2421641365,v1:172.21.15.203:6837/2421641365] conn(0x56278976d250 0x56278976d630 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x56278976ec70 0x56278978d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x56278979bee0 0x5627897829c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 --2- 172.21.15.124:0/1644081398 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56278976e890 0x5627897f9e90 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc8e17fa640 0 client.6048 ms_handle_reset on v2:172.21.15.203:6835/2421641365 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 >> 172.21.15.124:0/1644081398 conn(0x562789742000 msgr2=0x562789787780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 shutdown_connections 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.544+0000 7fc90b23e740 1 -- 172.21.15.124:0/1644081398 wait complete. 2023-12-13T23:57:22.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc90b23e740 20 client.6048 trim_cache size 0 max 16384 2023-12-13T23:57:22.553 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:22.553 INFO:teuthology.orchestra.run.smithi124.stdout: "cluster_fsid": "f45f9ea9-91cf-4589-b298-b4d7e1fc4f09", 2023-12-13T23:57:22.553 INFO:teuthology.orchestra.run.smithi124.stdout: "whoami": 0, 2023-12-13T23:57:22.553 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5910, 2023-12-13T23:57:22.553 INFO:teuthology.orchestra.run.smithi124.stdout: "want_state": "up:active", 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "state": "up:active", 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs", 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "rank_uptime": 4.5618231939999996, 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "mdsmap_epoch": 22, 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch": 50, 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "osdmap_epoch_barrier": 50, 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout: "uptime": 9.5804871820000006 2023-12-13T23:57:22.554 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 msgr2=0x7fc8e406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 0x7fc8e406d680 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc036000 tx=0x7fc8fc008030 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 msgr2=0x7fc9000caa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800cef0 tx=0x7fc8f800a710 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 shutdown_connections 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc8e406b1c0 0x7fc8e406d680 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc90015e730 0x7fc9000caa40 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc90015abf0 0x7fc90015c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 --2- 172.21.15.124:0/2945225621 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc900143490 0x7fc9001f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 >> 172.21.15.124:0/2945225621 conn(0x7fc90006a4d0 msgr2=0x7fc90006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 shutdown_connections 2023-12-13T23:57:22.557 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.545+0000 7fc9086a6640 1 -- 172.21.15.124:0/2945225621 wait complete. 2023-12-13T23:57:22.557 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:57:22.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 Processor -- start 2023-12-13T23:57:22.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 -- start start 2023-12-13T23:57:22.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f576815d330 0x7f57680b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 0x7f57680b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57680b2b10 con 0x7f57680d2910 2023-12-13T23:57:22.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57680d1590 con 0x7f576815d330 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.658+0000 7f576df33640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57680d1870 con 0x7f57680d3900 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f57677fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f576815d330 0x7f57680b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f5767fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 0x7f57680b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f575ffff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f57680d2910 0x7f57680d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:36030/0 (socket says 172.21.15.124:36030) 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f575ffff640 1 -- 172.21.15.124:0/725097135 learned_addr learned my addr 172.21.15.124:0/725097135 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f57677fe640 1 -- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 msgr2=0x7f57680b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f57677fe640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 0x7f57680b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f57677fe640 1 -- 172.21.15.124:0/725097135 >> v1:172.21.15.203:6789/0 conn(0x7f57680d2910 legacy=0x7f57680d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f57677fe640 1 -- 172.21.15.124:0/725097135 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57680d1b50 con 0x7f576815d330 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f5766ffd640 1 -- 172.21.15.124:0/725097135 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786080370 0 0) 0x7f57680b2b10 con 0x7f57680d2910 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.659+0000 7f5767fff640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 0x7f57680b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.660+0000 7f57677fe640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f576815d330 0x7f57680b8b80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f575802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9087dd474db7ba0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.660+0000 7f5766ffd640 1 -- 172.21.15.124:0/725097135 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575802e070 con 0x7f576815d330 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.660+0000 7f5766ffd640 1 -- 172.21.15.124:0/725097135 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57580315a0 con 0x7f576815d330 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.660+0000 7f5766ffd640 1 -- 172.21.15.124:0/725097135 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758031930 con 0x7f576815d330 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 -- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f576815d330 msgr2=0x7f57680b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f576815d330 0x7f57680b8b80 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f575802b840 comp rx=0 tx=0).stop 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 -- 172.21.15.124:0/725097135 shutdown_connections 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f57680d3900 0x7f57680b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 --2- 172.21.15.124:0/725097135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f576815d330 0x7f57680b8b80 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 -- 172.21.15.124:0/725097135 >> 172.21.15.124:0/725097135 conn(0x7f5768072020 msgr2=0x7f5768072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 -- 172.21.15.124:0/725097135 shutdown_connections 2023-12-13T23:57:22.662 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.661+0000 7f576df33640 1 -- 172.21.15.124:0/725097135 wait complete. 2023-12-13T23:57:22.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.662+0000 7f576df33640 1 Processor -- start 2023-12-13T23:57:22.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.662+0000 7f576df33640 1 -- start start 2023-12-13T23:57:22.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 0x7f576814e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f576815d330 0x7f576815fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680c7c60 con 0x7f57680d2910 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5768160820 con 0x7f57680d3900 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f576df33640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5768160540 con 0x7f576815d330 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57196/0 (socket says 172.21.15.124:57196) 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 -- 172.21.15.124:0/1167059935 learned_addr learned my addr 172.21.15.124:0/1167059935 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f5767fff640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f576815d330 0x7f576815fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f575ffff640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 0x7f576814e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 -- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f576815d330 msgr2=0x7f576815fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f576815d330 0x7f576815fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 -- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 msgr2=0x7f576814e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 0x7f576814e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 -- 172.21.15.124:0/1167059935 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5760009a70 con 0x7f57680d2910 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f575ffff640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 0x7f576814e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f57677fe640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f57580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.663+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758004230 con 0x7f57680d2910 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.664+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5758004390 con 0x7f57680d2910 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.664+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5758009400 con 0x7f57680d2910 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.664+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5758003850 con 0x7f57680d2910 2023-12-13T23:57:22.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.664+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57680c6f20 con 0x7f57680d2910 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.664+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5768069400 con 0x7f57680d2910 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.665+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f57580039b0 con 0x7f57680d2910 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.665+0000 7f5764ff9640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 0x7f574806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.665+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f57580ab900 con 0x7f57680d2910 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.665+0000 7f575ffff640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 0x7f574806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.666+0000 7f575ffff640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 0x7f574806d680 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f57681f6410 tx=0x7f5750007ca0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:22.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.669+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5758076630 con 0x7f57680d2910 2023-12-13T23:57:22.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.872+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f57680b3b30 con 0x7f57680d2910 2023-12-13T23:57:22.871 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.872+0000 7f5764ff9640 1 -- 172.21.15.124:0/1167059935 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5758079c50 con 0x7f57680d2910 2023-12-13T23:57:22.872 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 msgr2=0x7f574806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 0x7f574806d680 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f57681f6410 tx=0x7f5750007ca0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 msgr2=0x7f57680c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f57580097a0 tx=0x7f57580046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 shutdown_connections 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f574806b1c0 0x7f574806d680 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f57680d3900 0x7f576814e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f576815d330 0x7f576815fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 --2- 172.21.15.124:0/1167059935 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f57680d2910 0x7f57680c7720 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 >> 172.21.15.124:0/1167059935 conn(0x7f5768072020 msgr2=0x7f57680d31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:22.874 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 shutdown_connections 2023-12-13T23:57:22.875 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:22.874+0000 7f576df33640 1 -- 172.21.15.124:0/1167059935 wait complete. 2023-12-13T23:57:22.884 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:57:22.884 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:57:22.884 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:57:22.885 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:22.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:22.885 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:57:22.900 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:57:22.900 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:22.901 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42564sec preferred_lft 42564sec 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:22.902 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:22.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:22.903 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:57:22.966 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:57:22.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:22.968 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:57:23.023 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:57:23.023 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:57:23.023 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:57:23.023 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:57:23.023 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:57:23.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:57:23.024 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:57:23.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.118 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:57:23.133 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:57:23.133 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:57:23.133 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:23.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.133 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:57:23.200 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:57:23.200 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:23.200 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:23.254 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:23.255 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:23.255 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:23.309 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:23.377 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:23.434 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:23.435 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:23.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.436 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:23.488 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:57:23.488 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:57:23.531 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:23.558 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:23.615 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:23.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.616 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:23.617 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:57:23.616+0000 7f1da8640480 -1 init, newargv = 0x5633be006800 newargc=15 2023-12-13T23:57:23.618 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[108895]: starting ceph client 2023-12-13T23:57:23.620 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[108895]: starting fuse 2023-12-13T23:57:23.631 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:57:23.632 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:57:23.632 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:23.633 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:57:23.634 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:57:23.751 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.108895.asok 2023-12-13T23:57:23.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:23 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:57:23.761 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.108895.asok 2023-12-13T23:57:23.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.762 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.108895.asok status 2023-12-13T23:57:23.897 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:23.897 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:57:23.897 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:57:23.897 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "108895", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6057, 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6057 2023-12-13T23:57:23.898 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1692540320 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1692540320 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6057 192.168.0.1:0/1692540320", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/1692540320", 2023-12-13T23:57:23.899 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 22, 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 50, 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:57:23.900 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:23.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.906 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:57:23.906 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:23.922 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:57:23.922 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:23.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.922 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:23.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:23.990 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2023-12-13T23:57:24.052 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2023-12-13T23:57:24.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.199+0000 7f7b02092640 1 Processor -- start 2023-12-13T23:57:24.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.199+0000 7f7b02092640 1 -- start start 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7b02092640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7afc15b330 0x7f7afc071840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7b02092640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7afc0b0bf0 0x7f7afc0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7b02092640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7afc0b1580 con 0x7f7afc15e300 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7b02092640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc15b7c0 con 0x7f7afc15b330 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7b02092640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7afc15aea0 con 0x7f7afc0b0bf0 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7afaffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7afc15b330 0x7f7afc071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7afaffd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7afc15b330 0x7f7afc071840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57204/0 (socket says 172.21.15.124:57204) 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.200+0000 7f7afaffd640 1 -- 172.21.15.124:0/4260604065 learned_addr learned my addr 172.21.15.124:0/4260604065 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afbfff640 1 --2- 172.21.15.124:0/4260604065 >> v2:172.21.15.203:3300/0 conn(0x7f7afc0b0bf0 0x7f7afc0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afaffd640 1 -- 172.21.15.124:0/4260604065 >> v2:172.21.15.203:3300/0 conn(0x7f7afc0b0bf0 msgr2=0x7f7afc0b1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:24.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afaffd640 1 --2- 172.21.15.124:0/4260604065 >> v2:172.21.15.203:3300/0 conn(0x7f7afc0b0bf0 0x7f7afc0b1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afaffd640 1 -- 172.21.15.124:0/4260604065 >> v1:172.21.15.203:6789/0 conn(0x7f7afc15e300 legacy=0x7f7afc15e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:24.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afaffd640 1 -- 172.21.15.124:0/4260604065 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc072090 con 0x7f7afc15b330 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afaffd640 1 --2- 172.21.15.124:0/4260604065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc071840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7afc06e840 tx=0x7f7af002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546d6628e2e7e00d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afa7fc640 1 -- 172.21.15.124:0/4260604065 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0031870 con 0x7f7afc15b330 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.201+0000 7f7afa7fc640 1 -- 172.21.15.124:0/4260604065 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af00319d0 con 0x7f7afc15b330 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7afa7fc640 1 -- 172.21.15.124:0/4260604065 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0031d60 con 0x7f7afc15b330 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 -- 172.21.15.124:0/4260604065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 msgr2=0x7f7afc071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 --2- 172.21.15.124:0/4260604065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc071840 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f7afc06e840 tx=0x7f7af002b910 comp rx=0 tx=0).stop 2023-12-13T23:57:24.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 -- 172.21.15.124:0/4260604065 shutdown_connections 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 --2- 172.21.15.124:0/4260604065 >> v2:172.21.15.203:3300/0 conn(0x7f7afc0b0bf0 0x7f7afc0b1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 --2- 172.21.15.124:0/4260604065 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc071840 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 -- 172.21.15.124:0/4260604065 >> 172.21.15.124:0/4260604065 conn(0x7f7afc06d9c0 msgr2=0x7f7afc06a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 -- 172.21.15.124:0/4260604065 shutdown_connections 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.202+0000 7f7b02092640 1 -- 172.21.15.124:0/4260604065 wait complete. 2023-12-13T23:57:24.203 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.203+0000 7f7b02092640 1 Processor -- start 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.203+0000 7f7b02092640 1 -- start start 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 0x7f7afc13f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7afc15e300 0x7f7afc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc152f60 con 0x7f7afc15b330 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc13d090 con 0x7f7afc0b0bf0 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7b02092640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7afc1525c0 con 0x7f7afc15e300 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57210/0 (socket says 172.21.15.124:57210) 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 -- 172.21.15.124:0/1195436833 learned_addr learned my addr 172.21.15.124:0/1195436833 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afb7fe640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 0x7f7afc13f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 -- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7afc15e300 msgr2=0x7f7afc0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7afc15e300 0x7f7afc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 -- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 msgr2=0x7f7afc13f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 0x7f7afc13f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afaffd640 1 -- 172.21.15.124:0/1195436833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af00094d0 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.204+0000 7f7afb7fe640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 0x7f7afc13f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7afaffd640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7af0000c00 tx=0x7f7af00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af003abe0 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af003ad40 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7afc1528a0 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0044610 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.205+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7afc151490 con 0x7f7afc15b330 2023-12-13T23:57:24.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.206+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7afc1530a0 con 0x7f7afc15b330 2023-12-13T23:57:24.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.206+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7af004d450 con 0x7f7afc15b330 2023-12-13T23:57:24.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.207+0000 7f7ae3fff640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 0x7f7ae806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.207+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7af002d040 con 0x7f7afc15b330 2023-12-13T23:57:24.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.207+0000 7f7afb7fe640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 0x7f7ae806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.207+0000 7f7afb7fe640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 0x7f7ae806d6b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7aec003eb0 tx=0x7f7aec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.211+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7afc1530a0 con 0x7f7afc15b330 2023-12-13T23:57:24.443 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.444+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) v1 -- 0x7f7afc0b1580 con 0x7f7afc15b330 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.446+0000 7f7ae3fff640 1 -- 172.21.15.124:0/1195436833 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v16) v1 ==== 183+0+140 (secure 0 0 0) 0x7f7afc0b1580 con 0x7f7afc15b330 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:57:24.446 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow r pool=cephfs_data" 2023-12-13T23:57:24.447 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 msgr2=0x7f7ae806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 0x7f7ae806d6b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7aec003eb0 tx=0x7f7aec009040 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 msgr2=0x7f7afc150f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f7af0000c00 tx=0x7f7af00046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 shutdown_connections 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ae806b1f0 0x7f7ae806d6b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7afc15e300 0x7f7afc0b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7afc15b330 0x7f7afc150f50 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 --2- 172.21.15.124:0/1195436833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7afc0b0bf0 0x7f7afc13f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 >> 172.21.15.124:0/1195436833 conn(0x7f7afc06d9c0 msgr2=0x7f7afc15e6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 shutdown_connections 2023-12-13T23:57:24.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.448+0000 7f7b02092640 1 -- 172.21.15.124:0/1195436833 wait complete. 2023-12-13T23:57:24.458 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:24.458 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:24.477 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:24.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.612+0000 7f4fbf997640 1 Processor -- start 2023-12-13T23:57:24.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.612+0000 7f4fbf997640 1 -- start start 2023-12-13T23:57:24.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4fb8143490 0x7f4fb80b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 0x7f4fb80b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815a4e0 0x7f4fb815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.614 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb815b520 con 0x7f4fb8143490 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb8143870 con 0x7f4fb815e730 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbf997640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb815e2a0 con 0x7f4fb815a4e0 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbd733640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4fb8143490 0x7f4fb80b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbd733640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4fb8143490 0x7f4fb80b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57222/0 (socket says 172.21.15.124:57222) 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.613+0000 7f4fbd733640 1 -- 172.21.15.124:0/3474803445 learned_addr learned my addr 172.21.15.124:0/3474803445 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbcf32640 1 --2- 172.21.15.124:0/3474803445 >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 0x7f4fb80b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 -- 172.21.15.124:0/3474803445 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815a4e0 msgr2=0x7f4fb815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbdf34640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815a4e0 0x7f4fb815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815a4e0 0x7f4fb815a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 -- 172.21.15.124:0/3474803445 >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 msgr2=0x7f4fb80b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 --2- 172.21.15.124:0/3474803445 >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 0x7f4fb80b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 -- 172.21.15.124:0/3474803445 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb815b240 con 0x7f4fb8143490 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbcf32640 1 --2- 172.21.15.124:0/3474803445 >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 0x7f4fb80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:24.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.614+0000 7f4fbd733640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb8143490 0x7f4fb80b0cf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80aeca0 tx=0x7f4fb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b6f7ab7fa1105e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4faffff640 1 -- 172.21.15.124:0/3474803445 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb402ca70 con 0x7f4fb8143490 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4faffff640 1 -- 172.21.15.124:0/3474803445 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb402cbd0 con 0x7f4fb8143490 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4faffff640 1 -- 172.21.15.124:0/3474803445 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb40343f0 con 0x7f4fb8143490 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 -- 172.21.15.124:0/3474803445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb8143490 msgr2=0x7f4fb80b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb8143490 0x7f4fb80b0cf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f4fb80aeca0 tx=0x7f4fb402bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 -- 172.21.15.124:0/3474803445 shutdown_connections 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815a4e0 0x7f4fb815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 --2- 172.21.15.124:0/3474803445 >> v2:172.21.15.203:3300/0 conn(0x7f4fb815e730 0x7f4fb80b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 --2- 172.21.15.124:0/3474803445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb8143490 0x7f4fb80b0cf0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 -- 172.21.15.124:0/3474803445 >> 172.21.15.124:0/3474803445 conn(0x7f4fb806a4d0 msgr2=0x7f4fb806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 -- 172.21.15.124:0/3474803445 shutdown_connections 2023-12-13T23:57:24.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.615+0000 7f4fbf997640 1 -- 172.21.15.124:0/3474803445 wait complete. 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.616+0000 7f4fbf997640 1 Processor -- start 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.616+0000 7f4fbf997640 1 -- start start 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb8143490 0x7f4fb813e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815e730 0x7f4fb81415b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb813ee40 con 0x7f4fb815a4e0 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb81f0be0 con 0x7f4fb8143490 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbf997640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb80ab180 con 0x7f4fb815e730 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57230/0 (socket says 172.21.15.124:57230) 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 -- 172.21.15.124:0/1194800974 learned_addr learned my addr 172.21.15.124:0/1194800974 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:24.619 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbdf34640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815e730 0x7f4fb81415b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 -- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815e730 msgr2=0x7f4fb81415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815e730 0x7f4fb81415b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 -- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb8143490 msgr2=0x7f4fb813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb8143490 0x7f4fb813e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.617+0000 7f4fbcf32640 1 -- 172.21.15.124:0/1194800974 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb4009b30 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fbcf32640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8007820 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fa8007980 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb80ab460 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fa8007cf0 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.618+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb80aba20 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.619+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb80abc40 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.619+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f4fa80113f0 con 0x7f4fb815a4e0 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.620+0000 7f4fadffb640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 0x7f4f9c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.620 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.620+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4fa8089c00 con 0x7f4fb815a4e0 2023-12-13T23:57:24.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.620+0000 7f4fbd733640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 0x7f4f9c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.621 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.621+0000 7f4fbd733640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 0x7f4f9c06d680 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4035000 tx=0x7f4fb402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.623 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.624+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f4fa8090080 con 0x7f4fb815a4e0 2023-12-13T23:57:24.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.791+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4fb8069400 con 0x7f4fb815a4e0 2023-12-13T23:57:24.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.792+0000 7f4fadffb640 1 -- 172.21.15.124:0/1194800974 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f4fa801f090 con 0x7f4fb815a4e0 2023-12-13T23:57:24.792 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:24.792 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 msgr2=0x7f4f9c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 0x7f4f9c06d680 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f4fb4035000 tx=0x7f4fb402e000 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 msgr2=0x7f4fb80aa6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f4fa800c7f0 tx=0x7f4fa800ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 shutdown_connections 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f4f9c06b1c0 0x7f4f9c06d680 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f4fb815e730 0x7f4fb81415b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815a4e0 0x7f4fb80aa6e0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 --2- 172.21.15.124:0/1194800974 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f4fb8143490 0x7f4fb813e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 >> 172.21.15.124:0/1194800974 conn(0x7f4fb806a4d0 msgr2=0x7f4fb80b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:24.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 shutdown_connections 2023-12-13T23:57:24.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.794+0000 7f4fbf997640 1 -- 172.21.15.124:0/1194800974 wait complete. 2023-12-13T23:57:24.804 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:24.805 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:24.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.908+0000 7f7371b0f640 1 Processor -- start 2023-12-13T23:57:24.907 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.908+0000 7f7371b0f640 1 -- start start 2023-12-13T23:57:24.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f7371b0f640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f736c15e730 0x7f736c15abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.908 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f7371b0f640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736c15e2a0 con 0x7f736c143490 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f7371b0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f736c15b240 con 0x7f736c0b1400 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f7371b0f640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f736c15b520 con 0x7f736c15e730 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f736c15e730 0x7f736c15abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736b7fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f736c143490 0x7f736c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59210/0 (socket says 172.21.15.124:59210) 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736b7fe640 1 -- 172.21.15.124:0/796396063 learned_addr learned my addr 172.21.15.124:0/796396063 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736bfff640 1 -- 172.21.15.124:0/796396063 >> v1:172.21.15.203:6789/0 conn(0x7f736c0b1400 legacy=0x7f736c15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736bfff640 1 -- 172.21.15.124:0/796396063 >> v1:172.21.15.124:6789/0 conn(0x7f736c143490 legacy=0x7f736c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736bfff640 1 -- 172.21.15.124:0/796396063 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c15dc50 con 0x7f736c15e730 2023-12-13T23:57:24.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.909+0000 7f736a7fc640 1 -- 172.21.15.124:0/796396063 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457396367 0 0) 0x7f736c15e2a0 con 0x7f736c143490 2023-12-13T23:57:24.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.910+0000 7f736bfff640 1 --2- 172.21.15.124:0/796396063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c15e730 0x7f736c15abf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f736400bfa0 tx=0x7f736402e140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b178a9900ba876db server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.910+0000 7f736a7fc640 1 -- 172.21.15.124:0/796396063 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736402eae0 con 0x7f736c15e730 2023-12-13T23:57:24.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.910+0000 7f736a7fc640 1 -- 172.21.15.124:0/796396063 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736402ec40 con 0x7f736c15e730 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.910+0000 7f736a7fc640 1 -- 172.21.15.124:0/796396063 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7364009540 con 0x7f736c15e730 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 -- 172.21.15.124:0/796396063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c15e730 msgr2=0x7f736c15abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 --2- 172.21.15.124:0/796396063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c15e730 0x7f736c15abf0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f736400bfa0 tx=0x7f736402e140 comp rx=0 tx=0).stop 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 -- 172.21.15.124:0/796396063 shutdown_connections 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 --2- 172.21.15.124:0/796396063 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c15e730 0x7f736c15abf0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 -- 172.21.15.124:0/796396063 >> 172.21.15.124:0/796396063 conn(0x7f736c06a4d0 msgr2=0x7f736c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 -- 172.21.15.124:0/796396063 shutdown_connections 2023-12-13T23:57:24.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.911+0000 7f7371b0f640 1 -- 172.21.15.124:0/796396063 wait complete. 2023-12-13T23:57:24.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.912+0000 7f7371b0f640 1 Processor -- start 2023-12-13T23:57:24.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.912+0000 7f7371b0f640 1 -- start start 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 0x7f736c139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f736c15e730 0x7f736c140b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c13c490 con 0x7f736c143490 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c0ae4e0 con 0x7f736c0b1400 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736c15d2b0 con 0x7f736c15e730 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57248/0 (socket says 172.21.15.124:57248) 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 -- 172.21.15.124:0/2624048419 learned_addr learned my addr 172.21.15.124:0/2624048419 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736b7fe640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 0x7f736c139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736bfff640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f736c15e730 0x7f736c140b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 -- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f736c15e730 msgr2=0x7f736c140b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f736c15e730 0x7f736c140b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 -- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 msgr2=0x7f736c139af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 0x7f736c139af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 -- 172.21.15.124:0/2624048419 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736400bc80 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736b7fe640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 0x7f736c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f736affd640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f736000c450 tx=0x7f736000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73600052d0 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7360005430 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c13a030 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.913+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7360003830 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.914+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c13bed0 con 0x7f736c143490 2023-12-13T23:57:24.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.915+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c13bd30 con 0x7f736c143490 2023-12-13T23:57:24.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.915+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7360003990 con 0x7f736c143490 2023-12-13T23:57:24.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.915+0000 7f7353fff640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 0x7f735806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:24.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.915+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7360088fe0 con 0x7f736c143490 2023-12-13T23:57:24.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.916+0000 7f736b7fe640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 0x7f735806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:24.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.916+0000 7f736b7fe640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 0x7f735806d680 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f735c000f60 tx=0x7f735c007c50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:24.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:24.919+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f736c13bd30 con 0x7f736c143490 2023-12-13T23:57:25.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.091+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f736c13bd30 con 0x7f736c143490 2023-12-13T23:57:25.091 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.092+0000 7f7353fff640 1 -- 172.21.15.124:0/2624048419 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7360017020 con 0x7f736c143490 2023-12-13T23:57:25.091 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.093+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 msgr2=0x7f735806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 0x7f735806d680 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f735c000f60 tx=0x7f735c007c50 comp rx=0 tx=0).stop 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 msgr2=0x7f736c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f736000c450 tx=0x7f736000c910 comp rx=0 tx=0).stop 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 shutdown_connections 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f735806b1c0 0x7f735806d680 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f736c15e730 0x7f736c140b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f736c143490 0x7f736c0aeba0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 --2- 172.21.15.124:0/2624048419 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f736c0b1400 0x7f736c139af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 >> 172.21.15.124:0/2624048419 conn(0x7f736c06a4d0 msgr2=0x7f736c15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 shutdown_connections 2023-12-13T23:57:25.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.094+0000 7f7371b0f640 1 -- 172.21.15.124:0/2624048419 wait complete. 2023-12-13T23:57:25.104 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:25.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.207+0000 7f23fbecf640 1 Processor -- start 2023-12-13T23:57:25.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.207+0000 7f23fbecf640 1 -- start start 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fbecf640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23f40ad640 0x7f23f406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fbecf640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f415d720 con 0x7f23f406bbf0 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fbecf640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23f40adae0 con 0x7f23f415d340 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fbecf640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23f40addc0 con 0x7f23f40ad640 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23f946a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23f40ad640 0x7f23f406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fa46c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f23f406bbf0 0x7f23f406d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59212/0 (socket says 172.21.15.124:59212) 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23f946a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23f40ad640 0x7f23f406d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57262/0 (socket says 172.21.15.124:57262) 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.208+0000 7f23fa46c640 1 -- 172.21.15.124:0/4273588844 learned_addr learned my addr 172.21.15.124:0/4273588844 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f946a640 1 -- 172.21.15.124:0/4273588844 >> v1:172.21.15.203:6789/0 conn(0x7f23f415d340 legacy=0x7f23f406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f946a640 1 -- 172.21.15.124:0/4273588844 >> v1:172.21.15.124:6789/0 conn(0x7f23f406bbf0 legacy=0x7f23f406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f946a640 1 -- 172.21.15.124:0/4273588844 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f40ae0a0 con 0x7f23f40ad640 2023-12-13T23:57:25.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f946a640 1 --2- 172.21.15.124:0/4273588844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f40ad640 0x7f23f406d0b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097a0 tx=0x7f23e402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cceb73a2b83ef728 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f8c69640 1 -- 172.21.15.124:0/4273588844 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4030610 con 0x7f23f40ad640 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f8c69640 1 -- 172.21.15.124:0/4273588844 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e4030770 con 0x7f23f40ad640 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.209+0000 7f23f8c69640 1 -- 172.21.15.124:0/4273588844 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4030b00 con 0x7f23f40ad640 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 -- 172.21.15.124:0/4273588844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f40ad640 msgr2=0x7f23f406d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 --2- 172.21.15.124:0/4273588844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f40ad640 0x7f23f406d0b0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f23e40097a0 tx=0x7f23e402b960 comp rx=0 tx=0).stop 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 -- 172.21.15.124:0/4273588844 shutdown_connections 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 --2- 172.21.15.124:0/4273588844 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f40ad640 0x7f23f406d0b0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 -- 172.21.15.124:0/4273588844 >> 172.21.15.124:0/4273588844 conn(0x7f23f4070e70 msgr2=0x7f23f4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 -- 172.21.15.124:0/4273588844 shutdown_connections 2023-12-13T23:57:25.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.210+0000 7f23fbecf640 1 -- 172.21.15.124:0/4273588844 wait complete. 2023-12-13T23:57:25.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.211+0000 7f23fbecf640 1 Processor -- start 2023-12-13T23:57:25.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.211+0000 7f23fbecf640 1 -- start start 2023-12-13T23:57:25.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f406bbf0 0x7f23f40cde50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23f40ad640 0x7f23f41407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4140ce0 con 0x7f23f415d340 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f40cf120 con 0x7f23f406bbf0 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fbecf640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23f4143f30 con 0x7f23f40ad640 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57278/0 (socket says 172.21.15.124:57278) 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 -- 172.21.15.124:0/2384742673 learned_addr learned my addr 172.21.15.124:0/2384742673 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 -- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23f40ad640 msgr2=0x7f23f41407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23f40ad640 0x7f23f41407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 -- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f406bbf0 msgr2=0x7f23f40cde50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f406bbf0 0x7f23f40cde50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 -- 172.21.15.124:0/2384742673 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e4009400 con 0x7f23f415d340 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.212+0000 7f23fa46c640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c370 tx=0x7f23ec00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.213+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec005ce0 con 0x7f23f415d340 2023-12-13T23:57:25.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.213+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ec005e40 con 0x7f23f415d340 2023-12-13T23:57:25.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.213+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f40ce510 con 0x7f23f415d340 2023-12-13T23:57:25.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.213+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec0072d0 con 0x7f23f415d340 2023-12-13T23:57:25.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.213+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f40cead0 con 0x7f23f415d340 2023-12-13T23:57:25.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.214+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f4069400 con 0x7f23f415d340 2023-12-13T23:57:25.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.214+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f23ec0074f0 con 0x7f23f415d340 2023-12-13T23:57:25.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.214+0000 7f23ea7fc640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 0x7f23d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.214+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f23ec088e50 con 0x7f23f415d340 2023-12-13T23:57:25.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.215+0000 7f23f9c6b640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 0x7f23d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.215+0000 7f23f9c6b640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 0x7f23d806d680 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f23f00012e0 tx=0x7f23f0007d30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.218+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f23ec053c30 con 0x7f23f415d340 2023-12-13T23:57:25.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.385+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f23f41539b0 con 0x7f23f415d340 2023-12-13T23:57:25.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.385+0000 7f23ea7fc640 1 -- 172.21.15.124:0/2384742673 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f23ec017020 con 0x7f23f415d340 2023-12-13T23:57:25.385 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 msgr2=0x7f23d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 0x7f23d806d680 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f23f00012e0 tx=0x7f23f0007d30 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 msgr2=0x7f23f41439f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00c370 tx=0x7f23ec00c830 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 shutdown_connections 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23d806b1c0 0x7f23d806d680 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23f415d340 0x7f23f41439f0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f23f40ad640 0x7f23f41407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 --2- 172.21.15.124:0/2384742673 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f23f406bbf0 0x7f23f40cde50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 >> 172.21.15.124:0/2384742673 conn(0x7f23f4070e70 msgr2=0x7f23f406eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 shutdown_connections 2023-12-13T23:57:25.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.387+0000 7f23fbecf640 1 -- 172.21.15.124:0/2384742673 wait complete. 2023-12-13T23:57:25.397 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:25.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 Processor -- start 2023-12-13T23:57:25.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 -- start start 2023-12-13T23:57:25.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8ff40b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8ff40d2d90 0x7f8ff40b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff40b2b10 con 0x7f8ff40d2910 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff40d1590 con 0x7f8ff40d2d90 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.501+0000 7f8ffa69b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40d1870 con 0x7f8ff415d330 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff37fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8ff40d2910 0x7f8ff40d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59214/0 (socket says 172.21.15.124:59214) 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff37fe640 1 -- 172.21.15.124:0/1022004288 learned_addr learned my addr 172.21.15.124:0/1022004288 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff3fff640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8ff40b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/1022004288 >> v2:172.21.15.203:3300/0 conn(0x7f8ff40d2d90 0x7f8ff40b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002352083 0 0) 0x7f8ff40b2b10 con 0x7f8ff40d2910 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc003520 con 0x7f8ff40d2910 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4097448134 0 0) 0x7f8fdc003520 con 0x7f8ff40d2910 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 msgr2=0x7f8ff40b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8ff40b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 >> v2:172.21.15.203:3300/0 conn(0x7f8ff40d2d90 msgr2=0x7f8ff40b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 --2- 172.21.15.124:0/1022004288 >> v2:172.21.15.203:3300/0 conn(0x7f8ff40d2d90 0x7f8ff40b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff40d1b50 con 0x7f8ff40d2910 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8fe8001670 con 0x7f8ff40d2910 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 >> v1:172.21.15.124:6789/0 conn(0x7f8ff40d2910 legacy=0x7f8ff40d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fdc003970 0x7f8fdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8fdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fdc004d00 0x7f8fdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40d1590 con 0x7f8fdc004d00 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40d1870 con 0x7f8fdc003970 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fdc00b980 con 0x7f8ff415d330 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fdc004d00 0x7f8fdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff3fff640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fdc003970 0x7f8fdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.502+0000 7f8ff37fe640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8fdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 -- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 msgr2=0x7f8fdc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8fdc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 -- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fdc003970 msgr2=0x7f8fdc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8fdc003970 0x7f8fdc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 -- 172.21.15.124:0/1022004288 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc003520 con 0x7f8fdc004d00 2023-12-13T23:57:25.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff37fe640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff415d330 0x7f8fdc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:25.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fdc004d00 0x7f8fdc005180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009f90 tx=0x7f8fdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c8619bf1df1313d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec02cc20 con 0x7f8fdc004d00 2023-12-13T23:57:25.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00f040 con 0x7f8fdc004d00 2023-12-13T23:57:25.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.503+0000 7f8ff2ffd640 1 -- 172.21.15.124:0/1022004288 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec034440 con 0x7f8fdc004d00 2023-12-13T23:57:25.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 -- 172.21.15.124:0/1022004288 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fdc004d00 msgr2=0x7f8fdc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/1022004288 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8fdc004d00 0x7f8fdc005180 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009f90 tx=0x7f8fdc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ff8c38640 1 -- 172.21.15.124:0/1022004288 reap_dead start 2023-12-13T23:57:25.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 -- 172.21.15.124:0/1022004288 shutdown_connections 2023-12-13T23:57:25.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 -- 172.21.15.124:0/1022004288 >> 172.21.15.124:0/1022004288 conn(0x7f8ff4072020 msgr2=0x7f8ff4072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 -- 172.21.15.124:0/1022004288 shutdown_connections 2023-12-13T23:57:25.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.504+0000 7f8ffa69b640 1 -- 172.21.15.124:0/1022004288 wait complete. 2023-12-13T23:57:25.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.505+0000 7f8ffa69b640 1 Processor -- start 2023-12-13T23:57:25.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.505+0000 7f8ffa69b640 1 -- start start 2023-12-13T23:57:25.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff41548a0 0x7f8ff41559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.511 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 0x7f8ff4157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff41600d0 con 0x7f8ff4157a90 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff415f4b0 con 0x7f8ff40cd000 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ffa69b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff4155f10 con 0x7f8ff41548a0 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57290/0 (socket says 172.21.15.124:57290) 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 -- 172.21.15.124:0/3260948833 learned_addr learned my addr 172.21.15.124:0/3260948833 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff3fff640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff41548a0 0x7f8ff41559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff37fe640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 0x7f8ff4157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 -- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff41548a0 msgr2=0x7f8ff41559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff41548a0 0x7f8ff41559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 -- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 msgr2=0x7f8ff4157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 0x7f8ff4157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.506+0000 7f8ff8c38640 1 -- 172.21.15.124:0/3260948833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff41604f0 con 0x7f8ff4157a90 2023-12-13T23:57:25.512 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ff37fe640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 0x7f8ff4157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ff8c38640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009f90 tx=0x7f8fec009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec034cd0 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec034e30 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff415f730 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec003bb0 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.507+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff415fcf0 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.508+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4069400 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.508+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8fec003d10 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.509+0000 7f8ff0ff9640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 0x7f8fe006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.509+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8fec0ab320 con 0x7f8ff4157a90 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.509+0000 7f8ff3fff640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 0x7f8fe006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.513 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.509+0000 7f8ff3fff640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 0x7f8fe006d680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009a80 tx=0x7f8fe4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.513+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8fec076100 con 0x7f8ff4157a90 2023-12-13T23:57:25.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.678+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8ff40c6100 con 0x7f8ff4157a90 2023-12-13T23:57:25.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.679+0000 7f8ff0ff9640 1 -- 172.21.15.124:0/3260948833 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8fec079720 con 0x7f8ff4157a90 2023-12-13T23:57:25.678 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.680+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 msgr2=0x7f8fe006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.680+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 0x7f8fe006d680 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8fe4009a80 tx=0x7f8fe4008040 comp rx=0 tx=0).stop 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.680+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 msgr2=0x7f8ff41f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009f90 tx=0x7f8fec009750 comp rx=0 tx=0).stop 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 shutdown_connections 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8fe006b1c0 0x7f8fe006d680 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8ff40cd000 0x7f8ff4157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8ff4157a90 0x7f8ff41f63b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 --2- 172.21.15.124:0/3260948833 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8ff41548a0 0x7f8ff41559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 >> 172.21.15.124:0/3260948833 conn(0x7f8ff4072020 msgr2=0x7f8ff40b40d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 shutdown_connections 2023-12-13T23:57:25.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.681+0000 7f8ffa69b640 1 -- 172.21.15.124:0/3260948833 wait complete. 2023-12-13T23:57:25.690 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:57:25.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.794+0000 7ff62969b640 1 Processor -- start 2023-12-13T23:57:25.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.794+0000 7ff62969b640 1 -- start start 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff62969b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff6240ad640 0x7ff62406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff62969b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240adae0 con 0x7ff62415d1d0 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff62969b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240addc0 con 0x7ff62406bbf0 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff62969b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6240ae0a0 con 0x7ff6240ad640 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff622ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ff62415d1d0 0x7ff62406b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59220/0 (socket says 172.21.15.124:59220) 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff622ffd640 1 -- 172.21.15.124:0/1447656149 learned_addr learned my addr 172.21.15.124:0/1447656149 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741107768 0 0) 0x7ff6240adae0 con 0x7ff62415d1d0 2023-12-13T23:57:25.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60c003520 con 0x7ff62415d1d0 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1447656149 >> v2:172.21.15.203:3300/0 conn(0x7ff6240ad640 0x7ff62406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3080081412 0 0) 0x7ff60c003520 con 0x7ff62415d1d0 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 >> v2:172.21.15.203:3300/0 conn(0x7ff6240ad640 msgr2=0x7ff62406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 --2- 172.21.15.124:0/1447656149 >> v2:172.21.15.203:3300/0 conn(0x7ff6240ad640 0x7ff62406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 >> v1:172.21.15.203:6789/0 conn(0x7ff62406bbf0 legacy=0x7ff62406d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240ae380 con 0x7ff62415d1d0 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7ff6140019f0 con 0x7ff62415d1d0 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 >> v1:172.21.15.124:6789/0 conn(0x7ff62415d1d0 legacy=0x7ff62406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff60c003970 0x7ff60c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c005180 0x7ff60c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240ae0a0 con 0x7ff60c0047c0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c00c020 con 0x7ff60c005180 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.795+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c00c1e0 con 0x7ff60c003970 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff622ffd640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff60c003970 0x7ff60c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6237fe640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c005180 0x7ff60c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 -- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff60c003970 msgr2=0x7ff60c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff60c003970 0x7ff60c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 -- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c005180 msgr2=0x7ff60c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c005180 0x7ff60c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 -- 172.21.15.124:0/1447656149 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c003520 con 0x7ff60c0047c0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6237fe640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c005180 0x7ff60c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6237fe640 1 -- 172.21.15.124:0/1447656149 reap_dead start 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.796+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff618009ac0 tx=0x7ff60c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47359b1b041d056 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61802c6b0 con 0x7ff60c0047c0 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61802c810 con 0x7ff60c0047c0 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff621ffb640 1 -- 172.21.15.124:0/1447656149 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61802cba0 con 0x7ff60c0047c0 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 -- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 msgr2=0x7ff60c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff618009ac0 tx=0x7ff60c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 -- 172.21.15.124:0/1447656149 shutdown_connections 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 --2- 172.21.15.124:0/1447656149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff60c0047c0 0x7ff60c004c40 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 -- 172.21.15.124:0/1447656149 >> 172.21.15.124:0/1447656149 conn(0x7ff624070e70 msgr2=0x7ff624071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 -- 172.21.15.124:0/1447656149 shutdown_connections 2023-12-13T23:57:25.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.797+0000 7ff62969b640 1 -- 172.21.15.124:0/1447656149 wait complete. 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.798+0000 7ff62969b640 1 Processor -- start 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.798+0000 7ff62969b640 1 -- start start 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 0x7ff62415f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 0x7ff6240cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cf350 con 0x7ff6241f5b10 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cd0a0 con 0x7ff60c0047c0 2023-12-13T23:57:25.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff62969b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240cd380 con 0x7ff6240ce990 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff6237fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff6237fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57316/0 (socket says 172.21.15.124:57316) 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff6237fe640 1 -- 172.21.15.124:0/1646536487 learned_addr learned my addr 172.21.15.124:0/1646536487 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 0x7ff6240cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.799+0000 7ff622ffd640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 0x7ff62415f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 -- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 msgr2=0x7ff6240cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 0x7ff6240cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 -- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 msgr2=0x7ff62415f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 0x7ff62415f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 -- 172.21.15.124:0/1646536487 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240cd660 con 0x7ff6241f5b10 2023-12-13T23:57:25.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6227fc640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 0x7ff6240cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff622ffd640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 0x7ff62415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff6237fe640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff6240b4200 tx=0x7ff61c00e5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c0050d0 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c005230 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6240cd940 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61c0035f0 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.800+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6241f64d0 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.801+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6240cde60 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.801+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff61c003750 con 0x7ff6241f5b10 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.802+0000 7ff60b7fe640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 0x7ff61006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:25.805 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.802+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff61c088f50 con 0x7ff6241f5b10 2023-12-13T23:57:25.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.802+0000 7ff622ffd640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 0x7ff61006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:25.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.802+0000 7ff622ffd640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 0x7ff61006d6b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff614002230 tx=0x7ff6140020e0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:25.806 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.806+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff6240cde60 con 0x7ff6241f5b10 2023-12-13T23:57:25.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.983+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff6240cde60 con 0x7ff6241f5b10 2023-12-13T23:57:25.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.983+0000 7ff60b7fe640 1 -- 172.21.15.124:0/1646536487 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff6240cde60 con 0x7ff6241f5b10 2023-12-13T23:57:25.982 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:57:25.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 msgr2=0x7ff61006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 0x7ff61006d6b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff614002230 tx=0x7ff6140020e0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 msgr2=0x7ff6241f5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:25.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff6240b4200 tx=0x7ff61c00e5f0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 shutdown_connections 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff61006b1f0 0x7ff61006d6b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff6241f5b10 0x7ff6241f5f90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff6240ce990 0x7ff6240cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 --2- 172.21.15.124:0/1646536487 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff60c0047c0 0x7ff62415f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 >> 172.21.15.124:0/1646536487 conn(0x7ff624070e70 msgr2=0x7ff62406d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 shutdown_connections 2023-12-13T23:57:25.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:25.985+0000 7ff62969b640 1 -- 172.21.15.124:0/1646536487 wait complete. 2023-12-13T23:57:25.995 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:25.995 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:26.010 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.152+0000 7f367b461640 1 Processor -- start 2023-12-13T23:57:26.152 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 -- start start 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740cd720 0x7f367415f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f36740b52f0 0x7f36740b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36741592e0 con 0x7f3674158f00 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367415c8c0 con 0x7f36740b52f0 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.153+0000 7f367b461640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367415fdf0 con 0x7f36740cd720 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f36740b52f0 0x7f36740b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f36740b52f0 0x7f36740b56f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:56284/0 (socket says 172.21.15.124:56284) 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 -- 172.21.15.124:0/3728538869 learned_addr learned my addr 172.21.15.124:0/3728538869 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:26.153 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36791fd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3674158f00 0x7f367415b920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:36078/0 (socket says 172.21.15.124:36078) 2023-12-13T23:57:26.154 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36789fc640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740cd720 0x7f367415f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 -- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740cd720 msgr2=0x7f367415f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740cd720 0x7f367415f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 -- 172.21.15.124:0/3728538869 >> v1:172.21.15.203:6789/0 conn(0x7f3674158f00 legacy=0x7f367415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.154+0000 7f36799fe640 1 -- 172.21.15.124:0/3728538869 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36740b5c30 con 0x7f36740b52f0 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.155+0000 7f36799fe640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740b52f0 0x7f36740b56f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f366c009e80 tx=0x7f366c02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3581d424e716fbe1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.155 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.155+0000 7f366bfff640 1 -- 172.21.15.124:0/3728538869 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c02c8a0 con 0x7f36740b52f0 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.155+0000 7f366bfff640 1 -- 172.21.15.124:0/3728538869 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c02ca00 con 0x7f36740b52f0 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.155+0000 7f366bfff640 1 -- 172.21.15.124:0/3728538869 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c02cd90 con 0x7f36740b52f0 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 -- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740b52f0 msgr2=0x7f36740b56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740b52f0 0x7f36740b56f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f366c009e80 tx=0x7f366c02bd80 comp rx=0 tx=0).stop 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 -- 172.21.15.124:0/3728538869 shutdown_connections 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740b52f0 0x7f36740b56f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 --2- 172.21.15.124:0/3728538869 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740cd720 0x7f367415f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 -- 172.21.15.124:0/3728538869 >> 172.21.15.124:0/3728538869 conn(0x7f36740cff30 msgr2=0x7f36740d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.156 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 -- 172.21.15.124:0/3728538869 shutdown_connections 2023-12-13T23:57:26.157 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.156+0000 7f367b461640 1 -- 172.21.15.124:0/3728538869 wait complete. 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.157+0000 7f367b461640 1 Processor -- start 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.157+0000 7f367b461640 1 -- start start 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740b52f0 0x7f3674157a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 0x7f36740c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367413eae0 con 0x7f3674158f00 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367415a680 con 0x7f36740cd720 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f367b461640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367415a960 con 0x7f36740b52f0 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.160 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57324/0 (socket says 172.21.15.124:57324) 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 -- 172.21.15.124:0/2920977023 learned_addr learned my addr 172.21.15.124:0/2920977023 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36791fd640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740b52f0 0x7f3674157a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36789fc640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 0x7f36740c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 -- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740b52f0 msgr2=0x7f3674157a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740b52f0 0x7f3674157a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 -- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 msgr2=0x7f36740c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 0x7f36740c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 -- 172.21.15.124:0/2920977023 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c009b30 con 0x7f3674158f00 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36789fc640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 0x7f36740c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.158+0000 7f36799fe640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f366c02bdc0 tx=0x7f366c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.159+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c009510 con 0x7f3674158f00 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.159+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c007930 con 0x7f3674158f00 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.159+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367415ac40 con 0x7f3674158f00 2023-12-13T23:57:26.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.159+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c007ce0 con 0x7f3674158f00 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.159+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3674157f80 con 0x7f3674158f00 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.160+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367413ec90 con 0x7f3674158f00 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.160+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f366c03e450 con 0x7f3674158f00 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.161+0000 7f3669ffb640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 0x7f365806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.161+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f366c0ac370 con 0x7f3674158f00 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.161+0000 7f36791fd640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 0x7f365806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.161+0000 7f36791fd640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 0x7f365806d680 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3670007fd0 tx=0x7f3670007c20 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.164 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.165+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f367413ec90 con 0x7f3674158f00 2023-12-13T23:57:26.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.331+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3674069400 con 0x7f3674158f00 2023-12-13T23:57:26.331 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.332+0000 7f3669ffb640 1 -- 172.21.15.124:0/2920977023 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f366c0770a0 con 0x7f3674158f00 2023-12-13T23:57:26.332 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:26.332 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:26.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 msgr2=0x7f365806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 0x7f365806d680 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3670007fd0 tx=0x7f3670007c20 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 msgr2=0x7f36740c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f366c02bdc0 tx=0x7f366c004a40 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 shutdown_connections 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f365806b1c0 0x7f365806d680 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f36740cd720 0x7f36740c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3674158f00 0x7f36740c98a0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 --2- 172.21.15.124:0/2920977023 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f36740b52f0 0x7f3674157a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 >> 172.21.15.124:0/2920977023 conn(0x7f36740cff30 msgr2=0x7f367415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 shutdown_connections 2023-12-13T23:57:26.334 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.334+0000 7f367b461640 1 -- 172.21.15.124:0/2920977023 wait complete. 2023-12-13T23:57:26.344 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:26.344 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:26.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 Processor -- start 2023-12-13T23:57:26.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 -- start start 2023-12-13T23:57:26.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f014415e730 0x7f014415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014415e2a0 con 0x7f01440b1400 2023-12-13T23:57:26.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014415b240 con 0x7f0144143490 2023-12-13T23:57:26.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.448+0000 7f014adf1640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014415b520 con 0x7f014415e730 2023-12-13T23:57:26.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f0143fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f01440b1400 0x7f014415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59234/0 (socket says 172.21.15.124:59234) 2023-12-13T23:57:26.451 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f0143fff640 1 -- 172.21.15.124:0/379795122 learned_addr learned my addr 172.21.15.124:0/379795122 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f014938e640 1 --2- 172.21.15.124:0/379795122 >> v2:172.21.15.203:3300/0 conn(0x7f014415e730 0x7f014415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315503999 0 0) 0x7f014415e2a0 con 0x7f01440b1400 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0134003520 con 0x7f01440b1400 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456563905 0 0) 0x7f0134003520 con 0x7f01440b1400 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 >> v2:172.21.15.203:3300/0 conn(0x7f014415e730 msgr2=0x7f014415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 --2- 172.21.15.124:0/379795122 >> v2:172.21.15.203:3300/0 conn(0x7f014415e730 0x7f014415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 >> v1:172.21.15.203:6789/0 conn(0x7f0144143490 legacy=0x7f01440b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014415dc50 con 0x7f01440b1400 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0130001670 con 0x7f01440b1400 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 >> v1:172.21.15.124:6789/0 conn(0x7f01440b1400 legacy=0x7f014415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f0134003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01340047c0 0x7f0134004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0134005180 0x7f013400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.452 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014415b520 con 0x7f0134003970 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013400c020 con 0x7f01340047c0 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013400c1e0 con 0x7f0134005180 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f0148b8d640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f0134003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.449+0000 7f0143fff640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01340047c0 0x7f0134004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f014938e640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0134005180 0x7f013400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 -- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0134005180 msgr2=0x7f013400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0134005180 0x7f013400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 -- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01340047c0 msgr2=0x7f0134004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01340047c0 0x7f0134004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 -- 172.21.15.124:0/379795122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0134003520 con 0x7f0134003970 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f014938e640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0134005180 0x7f013400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0143fff640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01340047c0 0x7f0134004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:26.453 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f014938e640 1 -- 172.21.15.124:0/379795122 reap_dead start 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f0148b8d640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f0134003d90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0138001750 tx=0x7f01340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d905ec7ec3bec8d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380052d0 con 0x7f0134003970 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013802e050 con 0x7f0134003970 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.450+0000 7f01437fe640 1 -- 172.21.15.124:0/379795122 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138003830 con 0x7f0134003970 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 -- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 msgr2=0x7f0134003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f0134003d90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f0138001750 tx=0x7f01340057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 -- 172.21.15.124:0/379795122 shutdown_connections 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 --2- 172.21.15.124:0/379795122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f0134003d90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.454 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 -- 172.21.15.124:0/379795122 >> 172.21.15.124:0/379795122 conn(0x7f014406a4d0 msgr2=0x7f014406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 -- 172.21.15.124:0/379795122 shutdown_connections 2023-12-13T23:57:26.455 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.451+0000 7f014adf1640 1 -- 172.21.15.124:0/379795122 wait complete. 2023-12-13T23:57:26.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.452+0000 7f014adf1640 1 Processor -- start 2023-12-13T23:57:26.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.452+0000 7f014adf1640 1 -- start start 2023-12-13T23:57:26.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.457 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01441e47c0 0x7f0144141890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0144159c30 0x7f0144158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01441e4330 con 0x7f0134003970 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014413de50 con 0x7f0144159c30 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014adf1640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014413db80 con 0x7f01441e47c0 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57340/0 (socket says 172.21.15.124:57340) 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 -- 172.21.15.124:0/4232386910 learned_addr learned my addr 172.21.15.124:0/4232386910 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f014938e640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0144159c30 0x7f0144158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0143fff640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01441e47c0 0x7f0144141890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 -- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01441e47c0 msgr2=0x7f0144141890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01441e47c0 0x7f0144141890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 -- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0144159c30 msgr2=0x7f0144158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0144159c30 0x7f0144158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.458 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.453+0000 7f0148b8d640 1 -- 172.21.15.124:0/4232386910 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01441e3a00 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f0148b8d640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f013802bc00 tx=0x7f0138001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01380076c0 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0138007a30 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01441e3ce0 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0138006be0 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.454+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0144141dd0 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.455+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144069400 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.455+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0138006d40 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.456+0000 7f01417fa640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 0x7f012806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.456+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01380ab5c0 con 0x7f0134003970 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.456+0000 7f0143fff640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 0x7f012806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.459 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.456+0000 7f0143fff640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 0x7f012806d6b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f014413e5f0 tx=0x7f0130009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.460 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.460+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0138076380 con 0x7f0134003970 2023-12-13T23:57:26.626 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.627+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f01440b2820 con 0x7f0134003970 2023-12-13T23:57:26.627 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.627+0000 7f01417fa640 1 -- 172.21.15.124:0/4232386910 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f01380799a0 con 0x7f0134003970 2023-12-13T23:57:26.627 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 msgr2=0x7f012806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 0x7f012806d6b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f014413e5f0 tx=0x7f0130009040 comp rx=0 tx=0).stop 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 msgr2=0x7f014413d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f013802bc00 tx=0x7f0138001120 comp rx=0 tx=0).stop 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 shutdown_connections 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f012806b1f0 0x7f012806d6b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0144159c30 0x7f0144158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01441e47c0 0x7f0144141890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.629 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 --2- 172.21.15.124:0/4232386910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0134003970 0x7f014413d640 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 >> 172.21.15.124:0/4232386910 conn(0x7f014406a4d0 msgr2=0x7f014406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 shutdown_connections 2023-12-13T23:57:26.630 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.629+0000 7f014adf1640 1 -- 172.21.15.124:0/4232386910 wait complete. 2023-12-13T23:57:26.639 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:26.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.743+0000 7fc532b9e640 1 Processor -- start 2023-12-13T23:57:26.742 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.743+0000 7fc532b9e640 1 -- start start 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 0x7fc52c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc52c0b4370 0x7fc52c0b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c0b4d20 con 0x7fc52c0b4370 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52c071770 con 0x7fc52c144980 2023-12-13T23:57:26.744 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc532b9e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0ccdc0 con 0x7fc52c0ceb70 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc52c0b4370 0x7fc52c0b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc52c0b4370 0x7fc52c0b47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57354/0 (socket says 172.21.15.124:57354) 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 -- 172.21.15.124:0/4228544360 learned_addr learned my addr 172.21.15.124:0/4228544360 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53093a640 1 --2- 172.21.15.124:0/4228544360 >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 0x7fc52c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc52bfff640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 -- 172.21.15.124:0/4228544360 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 msgr2=0x7fc52c0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 -- 172.21.15.124:0/4228544360 >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 msgr2=0x7fc52c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 --2- 172.21.15.124:0/4228544360 >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 0x7fc52c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53113b640 1 -- 172.21.15.124:0/4228544360 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52c0cd0a0 con 0x7fc52c0b4370 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.744+0000 7fc53093a640 1 --2- 172.21.15.124:0/4228544360 >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 0x7fc52c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:26.745 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.745+0000 7fc53113b640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b47e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0b4ec0 tx=0x7fc52402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7c5334ee8593d6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.745+0000 7fc52b7fe640 1 -- 172.21.15.124:0/4228544360 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52402ca70 con 0x7fc52c0b4370 2023-12-13T23:57:26.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.745+0000 7fc52b7fe640 1 -- 172.21.15.124:0/4228544360 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc52402cbd0 con 0x7fc52c0b4370 2023-12-13T23:57:26.746 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.745+0000 7fc52b7fe640 1 -- 172.21.15.124:0/4228544360 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5240343f0 con 0x7fc52c0b4370 2023-12-13T23:57:26.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 -- 172.21.15.124:0/4228544360 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 msgr2=0x7fc52c0b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b47e0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fc52c0b4ec0 tx=0x7fc52402bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 -- 172.21.15.124:0/4228544360 shutdown_connections 2023-12-13T23:57:26.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b47e0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.748 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 --2- 172.21.15.124:0/4228544360 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 --2- 172.21.15.124:0/4228544360 >> v2:172.21.15.203:3300/0 conn(0x7fc52c144980 0x7fc52c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 -- 172.21.15.124:0/4228544360 >> 172.21.15.124:0/4228544360 conn(0x7fc52c06a4d0 msgr2=0x7fc52c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 -- 172.21.15.124:0/4228544360 shutdown_connections 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.746+0000 7fc532b9e640 1 -- 172.21.15.124:0/4228544360 wait complete. 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.747+0000 7fc532b9e640 1 Processor -- start 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.747+0000 7fc532b9e640 1 -- start start 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c06c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc52c144980 0x7fc52c1392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0b1040 con 0x7fc52c0b4370 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0b1320 con 0x7fc52c144980 2023-12-13T23:57:26.749 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52c0b1600 con 0x7fc52c0ceb70 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57358/0 (socket says 172.21.15.124:57358) 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 -- 172.21.15.124:0/3240441154 learned_addr learned my addr 172.21.15.124:0/3240441154 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 -- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 msgr2=0x7fc52c06c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c06c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 -- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc52c144980 msgr2=0x7fc52c1392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc52c144980 0x7fc52c1392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 -- 172.21.15.124:0/3240441154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc524009b30 con 0x7fc52c0b4370 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc53093a640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc52000c7f0 tx=0x7fc52000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.750 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc520007a30 con 0x7fc52c0b4370 2023-12-13T23:57:26.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520007b90 con 0x7fc52c0b4370 2023-12-13T23:57:26.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc52c0b18e0 con 0x7fc52c0b4370 2023-12-13T23:57:26.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.748+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc52000ce40 con 0x7fc52c0b4370 2023-12-13T23:57:26.751 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.749+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc52c06ccd0 con 0x7fc52c0b4370 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.749+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc52c0b1ad0 con 0x7fc52c0b4370 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.750+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc5200113f0 con 0x7fc52c0b4370 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.750+0000 7fc5297fa640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 0x7fc51006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.750+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc520089ad0 con 0x7fc52c0b4370 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.751+0000 7fc52bfff640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 0x7fc51006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:26.752 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.751+0000 7fc52bfff640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 0x7fc51006d680 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc52c1f6240 tx=0x7fc518008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:26.753 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.754+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc520054890 con 0x7fc52c0b4370 2023-12-13T23:57:26.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.918+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc52c069400 con 0x7fc52c0b4370 2023-12-13T23:57:26.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.919+0000 7fc5297fa640 1 -- 172.21.15.124:0/3240441154 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc52001f020 con 0x7fc52c0b4370 2023-12-13T23:57:26.918 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 msgr2=0x7fc51006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 0x7fc51006d680 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc52c1f6240 tx=0x7fc518008040 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 msgr2=0x7fc52c0b0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fc52000c7f0 tx=0x7fc52000cbf0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 shutdown_connections 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc51006b1c0 0x7fc51006d680 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc52c144980 0x7fc52c1392a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc52c0ceb70 0x7fc52c06c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 --2- 172.21.15.124:0/3240441154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc52c0b4370 0x7fc52c0b0a60 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.920+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 >> 172.21.15.124:0/3240441154 conn(0x7fc52c06a4d0 msgr2=0x7fc52c143b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.921+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 shutdown_connections 2023-12-13T23:57:26.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:26.921+0000 7fc532b9e640 1 -- 172.21.15.124:0/3240441154 wait complete. 2023-12-13T23:57:26.930 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:27.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.037+0000 7f63f983d640 1 Processor -- start 2023-12-13T23:57:27.037 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.037+0000 7f63f983d640 1 -- start start 2023-12-13T23:57:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f983d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63f40cf320 0x7f63f40d01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f983d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f63f415f760 0x7f63f40cd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f983d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63f40b3f10 con 0x7f63f415b130 2023-12-13T23:57:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f983d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f415fd20 con 0x7f63f40cf320 2023-12-13T23:57:27.038 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f983d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63f40d0730 con 0x7f63f415f760 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f27fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63f40cf320 0x7f63f40d01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f27fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63f40cf320 0x7f63f40d01f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57364/0 (socket says 172.21.15.124:57364) 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f2ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f63f415b130 0x7f63f40cec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59238/0 (socket says 172.21.15.124:59238) 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.038+0000 7f63f27fc640 1 -- 172.21.15.124:0/3996473924 learned_addr learned my addr 172.21.15.124:0/3996473924 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f27fc640 1 -- 172.21.15.124:0/3996473924 >> v2:172.21.15.203:3300/0 conn(0x7f63f415f760 msgr2=0x7f63f40cd310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3996473924 >> v2:172.21.15.203:3300/0 conn(0x7f63f415f760 0x7f63f40cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f27fc640 1 -- 172.21.15.124:0/3996473924 >> v1:172.21.15.124:6789/0 conn(0x7f63f415b130 legacy=0x7f63f40cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f27fc640 1 -- 172.21.15.124:0/3996473924 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f40d0a50 con 0x7f63f40cf320 2023-12-13T23:57:27.039 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f1ffb640 1 -- 172.21.15.124:0/3996473924 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460711855 0 0) 0x7f63f40b3f10 con 0x7f63f415b130 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3996473924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f40d01f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f63f406eb70 tx=0x7f63e802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8916be984451ca1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.039+0000 7f63f1ffb640 1 -- 172.21.15.124:0/3996473924 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8030860 con 0x7f63f40cf320 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f1ffb640 1 -- 172.21.15.124:0/3996473924 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e80309c0 con 0x7f63f40cf320 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f1ffb640 1 -- 172.21.15.124:0/3996473924 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e8030d50 con 0x7f63f40cf320 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 -- 172.21.15.124:0/3996473924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 msgr2=0x7f63f40d01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.041 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 --2- 172.21.15.124:0/3996473924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f40d01f0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f63f406eb70 tx=0x7f63e802b9a0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 -- 172.21.15.124:0/3996473924 shutdown_connections 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 --2- 172.21.15.124:0/3996473924 >> v2:172.21.15.203:3300/0 conn(0x7f63f415f760 0x7f63f40cd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 --2- 172.21.15.124:0/3996473924 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f40d01f0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 -- 172.21.15.124:0/3996473924 >> 172.21.15.124:0/3996473924 conn(0x7f63f40b2c10 msgr2=0x7f63f40b2ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 -- 172.21.15.124:0/3996473924 shutdown_connections 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.040+0000 7f63f983d640 1 -- 172.21.15.124:0/3996473924 wait complete. 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.041+0000 7f63f983d640 1 Processor -- start 2023-12-13T23:57:27.042 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 -- start start 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 0x7f63f40c3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 0x7f63f41501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f413f270 con 0x7f63f40cf320 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41f4280 con 0x7f63f415f760 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f983d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63f41f4560 con 0x7f63f415b130 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f2ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f2ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57366/0 (socket says 172.21.15.124:57366) 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.042+0000 7f63f2ffd640 1 -- 172.21.15.124:0/3551914122 learned_addr learned my addr 172.21.15.124:0/3551914122 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:27.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f37fe640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 0x7f63f41501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 0x7f63f40c3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 -- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 msgr2=0x7f63f40c3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 0x7f63f40c3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 -- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 msgr2=0x7f63f41501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 0x7f63f41501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 -- 172.21.15.124:0/3551914122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e80094d0 con 0x7f63f40cf320 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f37fe640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 0x7f63f41501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 0x7f63f40c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f2ffd640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f63e400c4d0 tx=0x7f63e400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e40074e0 con 0x7f63f40cf320 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63e4007660 con 0x7f63f40cf320 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63f41f4840 con 0x7f63f40cf320 2023-12-13T23:57:27.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63e4004a20 con 0x7f63f40cf320 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.043+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63f41f4e00 con 0x7f63f40cf320 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.044+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63f4067fc0 con 0x7f63f40cf320 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.044+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f63e4004b80 con 0x7f63f40cf320 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.045+0000 7f63dbfff640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 0x7f63e006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.045+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f63e4088f00 con 0x7f63f40cf320 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.045+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 0x7f63e006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.045+0000 7f63f27fc640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 0x7f63e006d6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63e8036000 tx=0x7f63e803b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.049+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f63e4053c30 con 0x7f63f40cf320 2023-12-13T23:57:27.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.218+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63f4067420 con 0x7f63f40cf320 2023-12-13T23:57:27.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.218+0000 7f63dbfff640 1 -- 172.21.15.124:0/3551914122 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f63e4018020 con 0x7f63f40cf320 2023-12-13T23:57:27.218 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:27.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 msgr2=0x7f63e006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 0x7f63e006d6b0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f63e8036000 tx=0x7f63e803b040 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 msgr2=0x7f63f4145310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f63e400c4d0 tx=0x7f63e400c990 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 shutdown_connections 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f63e006b1f0 0x7f63e006d6b0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f63f415f760 0x7f63f41501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63f415b130 0x7f63f40c3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 --2- 172.21.15.124:0/3551914122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63f40cf320 0x7f63f4145310 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 >> 172.21.15.124:0/3551914122 conn(0x7f63f40b2c10 msgr2=0x7f63f415d790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 shutdown_connections 2023-12-13T23:57:27.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.220+0000 7f63f983d640 1 -- 172.21.15.124:0/3551914122 wait complete. 2023-12-13T23:57:27.230 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:57:27.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.330+0000 7fee0fb61640 1 Processor -- start 2023-12-13T23:57:27.330 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 -- start start 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee08150560 0x7fee080b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fee080b3390 0x7fee0815d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fee08150940 0x7fee0815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080b49f0 con 0x7fee080b3390 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080b3de0 con 0x7fee08150940 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.331+0000 7fee0fb61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080b40c0 con 0x7fee08150560 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fee080b3390 0x7fee0815d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fee080b3390 0x7fee0815d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57374/0 (socket says 172.21.15.124:57374) 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 -- 172.21.15.124:0/1912195898 learned_addr learned my addr 172.21.15.124:0/1912195898 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0e0fe640 1 --2- 172.21.15.124:0/1912195898 >> v2:172.21.15.203:3300/0 conn(0x7fee08150940 0x7fee0815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d8fd640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee08150560 0x7fee080b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 -- 172.21.15.124:0/1912195898 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee08150560 msgr2=0x7fee080b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:27.332 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee08150560 0x7fee080b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 -- 172.21.15.124:0/1912195898 >> v2:172.21.15.203:3300/0 conn(0x7fee08150940 msgr2=0x7fee0815d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/1912195898 >> v2:172.21.15.203:3300/0 conn(0x7fee08150940 0x7fee0815d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.332+0000 7fee0d0fc640 1 -- 172.21.15.124:0/1912195898 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080b43a0 con 0x7fee080b3390 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee0815d200 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fee080aec00 tx=0x7fedf802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32746a4da80d19f4 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0c8fb640 1 -- 172.21.15.124:0/1912195898 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8031640 con 0x7fee080b3390 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0c8fb640 1 -- 172.21.15.124:0/1912195898 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf80317a0 con 0x7fee080b3390 2023-12-13T23:57:27.333 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0c8fb640 1 -- 172.21.15.124:0/1912195898 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8031b30 con 0x7fee080b3390 2023-12-13T23:57:27.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 -- 172.21.15.124:0/1912195898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 msgr2=0x7fee0815d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee0815d200 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fee080aec00 tx=0x7fedf802b8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 -- 172.21.15.124:0/1912195898 shutdown_connections 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee0815d200 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 --2- 172.21.15.124:0/1912195898 >> v2:172.21.15.203:3300/0 conn(0x7fee08150940 0x7fee0815d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 --2- 172.21.15.124:0/1912195898 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee08150560 0x7fee080b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 -- 172.21.15.124:0/1912195898 >> 172.21.15.124:0/1912195898 conn(0x7fee0806a4d0 msgr2=0x7fee0806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 -- 172.21.15.124:0/1912195898 shutdown_connections 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.333+0000 7fee0fb61640 1 -- 172.21.15.124:0/1912195898 wait complete. 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 Processor -- start 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 -- start start 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee08150560 0x7fee081566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee081453a0 0x7fee08157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081466f0 con 0x7fee080b3390 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08156210 con 0x7fee08150560 2023-12-13T23:57:27.336 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0fb61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee08155320 con 0x7fee081453a0 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0d8fd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0d8fd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57382/0 (socket says 172.21.15.124:57382) 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.335+0000 7fee0d8fd640 1 -- 172.21.15.124:0/2356004858 learned_addr learned my addr 172.21.15.124:0/2356004858 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0e0fe640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee081453a0 0x7fee08157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 -- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee081453a0 msgr2=0x7fee08157aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee081453a0 0x7fee08157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 -- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee08150560 msgr2=0x7fee081566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee08150560 0x7fee081566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee08150560 0x7fee081566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 -- 172.21.15.124:0/2356004858 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf80094d0 con 0x7fee080b3390 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0d8fd640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fee00007ef0 tx=0x7fee0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00014620 con 0x7fee080b3390 2023-12-13T23:57:27.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00014780 con 0x7fee080b3390 2023-12-13T23:57:27.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00014b30 con 0x7fee080b3390 2023-12-13T23:57:27.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08155600 con 0x7fee080b3390 2023-12-13T23:57:27.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.336+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee08155bc0 con 0x7fee080b3390 2023-12-13T23:57:27.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.337+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee08069400 con 0x7fee080b3390 2023-12-13T23:57:27.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.338+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fee00014c90 con 0x7fee080b3390 2023-12-13T23:57:27.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.338+0000 7fedfe7fc640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 0x7fedec06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.338+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fee0008a330 con 0x7fee080b3390 2023-12-13T23:57:27.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.338+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 0x7fedec06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.339+0000 7fee0d0fc640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 0x7fedec06d6b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fedf8036000 tx=0x7fedf803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.342+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fee000550f0 con 0x7fee080b3390 2023-12-13T23:57:27.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.521+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fee08157530 con 0x7fee080b3390 2023-12-13T23:57:27.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.521+0000 7fedfe7fc640 1 -- 172.21.15.124:0/2356004858 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fee00058710 con 0x7fee080b3390 2023-12-13T23:57:27.521 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 msgr2=0x7fedec06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 0x7fedec06d6b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fedf8036000 tx=0x7fedf803c040 comp rx=0 tx=0).stop 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 msgr2=0x7fee081461b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fee00007ef0 tx=0x7fee0000e4b0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 shutdown_connections 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fedec06b1f0 0x7fedec06d6b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fee08150560 0x7fee081566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fee081453a0 0x7fee08157aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 --2- 172.21.15.124:0/2356004858 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fee080b3390 0x7fee081461b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 >> 172.21.15.124:0/2356004858 conn(0x7fee0806a4d0 msgr2=0x7fee080ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 shutdown_connections 2023-12-13T23:57:27.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.523+0000 7fee0fb61640 1 -- 172.21.15.124:0/2356004858 wait complete. 2023-12-13T23:57:27.533 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:57:27.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:27.534 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:27.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[108895]: fuse finished with error 0 and tester_r 0 2023-12-13T23:57:27.560 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:27.609 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:27.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:27.609 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:57:27.624 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.765+0000 7f0a5dd53640 1 Processor -- start 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.765+0000 7f0a5dd53640 1 -- start start 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a5dd53640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f0a5815e730 0x7f0a5815abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a5dd53640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a5815e2a0 con 0x7f0a580b1400 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a5dd53640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a5815b240 con 0x7f0a58143490 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a5dd53640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a5815b520 con 0x7f0a5815e730 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a56ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0a580b1400 0x7f0a5815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59244/0 (socket says 172.21.15.124:59244) 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a56ffd640 1 -- 172.21.15.124:0/1617451102 learned_addr learned my addr 172.21.15.124:0/1617451102 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a57fff640 1 --2- 172.21.15.124:0/1617451102 >> v2:172.21.15.203:3300/0 conn(0x7f0a5815e730 0x7f0a5815abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570325373 0 0) 0x7f0a5815e2a0 con 0x7f0a580b1400 2023-12-13T23:57:27.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a40003520 con 0x7f0a580b1400 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204831975 0 0) 0x7f0a5815b240 con 0x7f0a58143490 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.766+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5815e2a0 con 0x7f0a58143490 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.767+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2612264507 0 0) 0x7f0a40003520 con 0x7f0a580b1400 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.767+0000 7f0a57fff640 1 -- 172.21.15.124:0/1617451102 >> v1:172.21.15.203:6789/0 conn(0x7f0a58143490 legacy=0x7f0a580b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.767+0000 7f0a57fff640 1 -- 172.21.15.124:0/1617451102 >> v1:172.21.15.124:6789/0 conn(0x7f0a580b1400 legacy=0x7f0a5815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.767+0000 7f0a57fff640 1 -- 172.21.15.124:0/1617451102 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5815dc50 con 0x7f0a5815e730 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.767+0000 7f0a57fff640 1 --2- 172.21.15.124:0/1617451102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5815abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009d90 tx=0x7f0a5002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=630e7aa38af2cb9d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c001670 con 0x7f0a5815e730 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5002c940 con 0x7f0a5815e730 2023-12-13T23:57:27.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a567fc640 1 -- 172.21.15.124:0/1617451102 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5002ccb0 con 0x7f0a5815e730 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 -- 172.21.15.124:0/1617451102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 msgr2=0x7f0a5815abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/1617451102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5815abf0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009d90 tx=0x7f0a5002be10 comp rx=0 tx=0).stop 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 -- 172.21.15.124:0/1617451102 shutdown_connections 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/1617451102 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5815abf0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 -- 172.21.15.124:0/1617451102 >> 172.21.15.124:0/1617451102 conn(0x7f0a5806a4d0 msgr2=0x7f0a5806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 -- 172.21.15.124:0/1617451102 shutdown_connections 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.768+0000 7f0a5dd53640 1 -- 172.21.15.124:0/1617451102 wait complete. 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.769+0000 7f0a5dd53640 1 Processor -- start 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.769+0000 7f0a5dd53640 1 -- start start 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a58143490 0x7f0a5813e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5813e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5813be80 con 0x7f0a580b1400 2023-12-13T23:57:27.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5813c160 con 0x7f0a5815e730 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a5dd53640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5813c440 con 0x7f0a58143490 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a577fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57398/0 (socket says 172.21.15.124:57398) 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.770+0000 7f0a577fe640 1 -- 172.21.15.124:0/560436657 learned_addr learned my addr 172.21.15.124:0/560436657 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a56ffd640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a58143490 0x7f0a5813e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 -- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a58143490 msgr2=0x7f0a5813e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a58143490 0x7f0a5813e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a57fff640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5813e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 -- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 msgr2=0x7f0a5813e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5813e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 -- 172.21.15.124:0/560436657 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50009a70 con 0x7f0a580b1400 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a577fe640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c490 tx=0x7f0a4800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a480052d0 con 0x7f0a580b1400 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a48005430 con 0x7f0a580b1400 2023-12-13T23:57:27.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a5813c720 con 0x7f0a580b1400 2023-12-13T23:57:27.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a48012870 con 0x7f0a580b1400 2023-12-13T23:57:27.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.771+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a5813cb80 con 0x7f0a580b1400 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.772+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a581632e0 con 0x7f0a580b1400 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.772+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0a480129d0 con 0x7f0a580b1400 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.773+0000 7f0a3ffff640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 0x7f0a4406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.773+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a48089690 con 0x7f0a580b1400 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.773+0000 7f0a56ffd640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 0x7f0a4406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:27.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.774+0000 7f0a56ffd640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 0x7f0a4406d6b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005880 tx=0x7f0a4c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:27.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.777+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f0a581632e0 con 0x7f0a580b1400 2023-12-13T23:57:27.977 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.978+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0a581632e0 con 0x7f0a580b1400 2023-12-13T23:57:27.978 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.978+0000 7f0a3ffff640 1 -- 172.21.15.124:0/560436657 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0a581632e0 con 0x7f0a580b1400 2023-12-13T23:57:27.978 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:57:27.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 msgr2=0x7f0a4406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.979 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 0x7f0a4406d6b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f0a4c005880 tx=0x7f0a4c009040 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 msgr2=0x7f0a5813b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c490 tx=0x7f0a4800c950 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 shutdown_connections 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0a4406b1f0 0x7f0a4406d6b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0a5815e730 0x7f0a5813e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0a58143490 0x7f0a5813e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 --2- 172.21.15.124:0/560436657 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0a580b1400 0x7f0a5813b400 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 >> 172.21.15.124:0/560436657 conn(0x7f0a5806a4d0 msgr2=0x7f0a581428c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 shutdown_connections 2023-12-13T23:57:27.980 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:27.980+0000 7f0a5dd53640 1 -- 172.21.15.124:0/560436657 wait complete. 2023-12-13T23:57:27.990 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:57:27.990 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:57:27.990 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:57:27.991 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:27.991 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:27.991 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:57:28.006 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:57:28.006 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:57:28.006 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:57:28.007 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42559sec preferred_lft 42559sec 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.008 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:57:28.009 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:28.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.010 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:57:28.073 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:57:28.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.074 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:57:28.130 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:57:28.130 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:57:28.131 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:57:28.131 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:57:28.131 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:57:28.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.132 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:57:28.132 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:57:28.132 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:57:28.132 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:57:28.132 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:57:28.133 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:57:28.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.227 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:57:28.242 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:57:28.242 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:57:28.242 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:28.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.242 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:57:28.309 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:57:28.309 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:28.309 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:28.364 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:28.364 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:28.364 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:28.419 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:28.486 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:28.542 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:28.542 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:28.542 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:28.542 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:28.542 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:28.543 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:28.544 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:28.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.545 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:28.597 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:57:28.597 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:57:28.639 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:28.665 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:28.721 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:28.722 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:28.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.723 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:28.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:57:28.724+0000 7f47c8ad6480 -1 init, newargv = 0x565352196e50 newargc=15 2023-12-13T23:57:28.724 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[109859]: starting ceph client 2023-12-13T23:57:28.728 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[109859]: starting fuse 2023-12-13T23:57:28.738 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:57:28.739 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:57:28.739 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:57:28.740 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:57:28.741 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:28.741 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:57:28.741 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:57:28.858 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.109859.asok 2023-12-13T23:57:28.861 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:57:28.868 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.109859.asok 2023-12-13T23:57:28.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:28.868 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.109859.asok status 2023-12-13T23:57:29.003 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:29.003 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:57:29.003 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:57:29.003 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:57:29.003 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "109859", 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6141, 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6141 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:29.004 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1188116819 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1188116819 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6141 192.168.0.1:0/1188116819", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/1188116819", 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 22, 2023-12-13T23:57:29.005 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 50, 2023-12-13T23:57:29.006 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:57:29.006 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:57:29.006 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:57:29.006 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:29.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:29.012 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:57:29.012 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:29.027 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:57:29.028 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:29.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:29.028 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> stdin-killer -- python3 -c ' 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> import errno 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> try: 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> if False: 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> ret = os.read(fd, 1024) 2023-12-13T23:57:29.095 DEBUG:teuthology.orchestra.run.smithi124:> else: 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> os.write(fd, b'"'"'content'"'"') 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> except OSError as e: 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> if e.errno != errno.EPERM: 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> raise 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> else: 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("client does not check permission of data pool") 2023-12-13T23:57:29.096 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:57:29.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:57:29.212 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2023-12-13T23:57:29.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.360+0000 7fcedff78640 1 Processor -- start 2023-12-13T23:57:29.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.360+0000 7fcedff78640 1 -- start start 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 0x7fced815ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 0x7fced80b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fced80b1130 0x7fced815ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced80b15c0 con 0x7fced80b1130 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced80ccdb0 con 0x7fced8158c60 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced815c730 con 0x7fced8159040 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fced80b1130 0x7fced815ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fced80b1130 0x7fced815ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57414/0 (socket says 172.21.15.124:57414) 2023-12-13T23:57:29.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 -- 172.21.15.124:0/1542901757 learned_addr learned my addr 172.21.15.124:0/1542901757 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fceddd14640 1 --2- 172.21.15.124:0/1542901757 >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 0x7fced815ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedd513640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 0x7fced80b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 -- 172.21.15.124:0/1542901757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 msgr2=0x7fced80b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 0x7fced80b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 -- 172.21.15.124:0/1542901757 >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 msgr2=0x7fced815ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 --2- 172.21.15.124:0/1542901757 >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 0x7fced815ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 -- 172.21.15.124:0/1542901757 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced80cd070 con 0x7fced80b1130 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fceddd14640 1 --2- 172.21.15.124:0/1542901757 >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 0x7fced815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedd513640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 0x7fced80b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcede515640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced80b1130 0x7fced815ba00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fced0009e50 tx=0x7fced002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33edda4f3978820c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedcd12640 1 -- 172.21.15.124:0/1542901757 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced002ca70 con 0x7fced80b1130 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedcd12640 1 -- 172.21.15.124:0/1542901757 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced002cbd0 con 0x7fced80b1130 2023-12-13T23:57:29.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedcd12640 1 -- 172.21.15.124:0/1542901757 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced00343f0 con 0x7fced80b1130 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 -- 172.21.15.124:0/1542901757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced80b1130 msgr2=0x7fced815ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.362+0000 7fcedff78640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced80b1130 0x7fced815ba00 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fced0009e50 tx=0x7fced002bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 -- 172.21.15.124:0/1542901757 shutdown_connections 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced80b1130 0x7fced815ba00 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 --2- 172.21.15.124:0/1542901757 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced8159040 0x7fced80b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 --2- 172.21.15.124:0/1542901757 >> v2:172.21.15.203:3300/0 conn(0x7fced8158c60 0x7fced815ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 -- 172.21.15.124:0/1542901757 >> 172.21.15.124:0/1542901757 conn(0x7fced806d9c0 msgr2=0x7fced806a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 -- 172.21.15.124:0/1542901757 shutdown_connections 2023-12-13T23:57:29.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.363+0000 7fcedff78640 1 -- 172.21.15.124:0/1542901757 wait complete. 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 Processor -- start 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 -- start start 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced80b1130 0x7fced806cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fced8159040 0x7fced814fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced80b0ff0 con 0x7fced8158c60 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced814f550 con 0x7fced8159040 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.364+0000 7fcedff78640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced814ccb0 con 0x7fced80b1130 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57426/0 (socket says 172.21.15.124:57426) 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 -- 172.21.15.124:0/4179973628 learned_addr learned my addr 172.21.15.124:0/4179973628 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fceddd14640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced80b1130 0x7fced806cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcede515640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fced8159040 0x7fced814fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 -- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced80b1130 msgr2=0x7fced806cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced80b1130 0x7fced806cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 -- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fced8159040 msgr2=0x7fced814fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fced8159040 0x7fced814fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 -- 172.21.15.124:0/4179973628 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0009b30 con 0x7fced8158c60 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedd513640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcec800d800 tx=0x7fcec800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8004030 con 0x7fced8158c60 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcec8004190 con 0x7fced8158c60 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced81537d0 con 0x7fced8158c60 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcec8004520 con 0x7fced8158c60 2023-12-13T23:57:29.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.365+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced8153d90 con 0x7fced8158c60 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.366+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced8153fb0 con 0x7fced8158c60 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.367+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fcec80096a0 con 0x7fced8158c60 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.367+0000 7fcece7fc640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 0x7fcebc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.367+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcec80899b0 con 0x7fced8158c60 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.367+0000 7fceddd14640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 0x7fcebc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.368+0000 7fceddd14640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 0x7fcebc06d680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fced4009950 tx=0x7fced4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.371+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fcec8054820 con 0x7fced8158c60 2023-12-13T23:57:29.604 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.604+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) v1 -- 0x7fced815aa20 con 0x7fced8158c60 2023-12-13T23:57:29.606 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.606+0000 7fcece7fc640 1 -- 172.21.15.124:0/4179973628 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v17) v1 ==== 183+0+140 (secure 0 0 0) 0x7fcec8057e40 con 0x7fced8158c60 2023-12-13T23:57:29.606 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:57:29.607 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:57:29.607 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:57:29.607 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:57:29.607 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow w pool=cephfs_data" 2023-12-13T23:57:29.607 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:57:29.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 msgr2=0x7fcebc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.609 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 0x7fcebc06d680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fced4009950 tx=0x7fced4008040 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 msgr2=0x7fced814c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcec800d800 tx=0x7fcec800dcc0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 shutdown_connections 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fcebc06b1c0 0x7fcebc06d680 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fced8159040 0x7fced814fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fced8158c60 0x7fced814c420 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 --2- 172.21.15.124:0/4179973628 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fced80b1130 0x7fced806cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 >> 172.21.15.124:0/4179973628 conn(0x7fced806d9c0 msgr2=0x7fced8159f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 shutdown_connections 2023-12-13T23:57:29.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.609+0000 7fcedff78640 1 -- 172.21.15.124:0/4179973628 wait complete. 2023-12-13T23:57:29.621 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:29.621 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:29.637 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 Processor -- start 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 -- start start 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b000b2b10 con 0x7f6b0015d330 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b000d1590 con 0x7f6b000d2910 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.777+0000 7f6b066c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b000d1870 con 0x7f6b000d3730 2023-12-13T23:57:29.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6aff7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:43640/0 (socket says 172.21.15.124:43640) 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6aff7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:56402/0 (socket says 172.21.15.124:56402) 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 -- 172.21.15.124:0/3145923370 learned_addr learned my addr 172.21.15.124:0/3145923370 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6afffff640 1 --1- 172.21.15.124:0/3145923370 >> v1:172.21.15.203:6789/0 conn(0x7f6b0015d330 0x7f6b000b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:36086/0 (socket says 172.21.15.124:36086) 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 -- 172.21.15.124:0/3145923370 >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 msgr2=0x7f6b000d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 --2- 172.21.15.124:0/3145923370 >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 -- 172.21.15.124:0/3145923370 >> v1:172.21.15.203:6789/0 conn(0x7f6b0015d330 legacy=0x7f6b000b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6afeffd640 1 -- 172.21.15.124:0/3145923370 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2570864671 0 0) 0x7f6b000b2b10 con 0x7f6b0015d330 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6b04c66640 1 -- 172.21.15.124:0/3145923370 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b000d3b90 con 0x7f6b000d3730 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.778+0000 7f6aff7fe640 1 --2- 172.21.15.124:0/3145923370 >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:29.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.779+0000 7f6b04c66640 1 --2- 172.21.15.124:0/3145923370 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e50 tx=0x7f6af802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4141b88d7e9d682 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.779+0000 7f6afeffd640 1 -- 172.21.15.124:0/3145923370 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af802ca70 con 0x7f6b000d3730 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.779+0000 7f6afeffd640 1 -- 172.21.15.124:0/3145923370 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af802cbd0 con 0x7f6b000d3730 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.779+0000 7f6afeffd640 1 -- 172.21.15.124:0/3145923370 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af80343f0 con 0x7f6b000d3730 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 -- 172.21.15.124:0/3145923370 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 msgr2=0x7f6b000b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 --2- 172.21.15.124:0/3145923370 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6af8009e50 tx=0x7f6af802bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 -- 172.21.15.124:0/3145923370 shutdown_connections 2023-12-13T23:57:29.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 --2- 172.21.15.124:0/3145923370 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b6390 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 --2- 172.21.15.124:0/3145923370 >> v2:172.21.15.203:3300/0 conn(0x7f6b000d2910 0x7f6b000d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 -- 172.21.15.124:0/3145923370 >> 172.21.15.124:0/3145923370 conn(0x7f6b00072020 msgr2=0x7f6b00072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:29.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 -- 172.21.15.124:0/3145923370 shutdown_connections 2023-12-13T23:57:29.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.780+0000 7f6b066c9640 1 -- 172.21.15.124:0/3145923370 wait complete. 2023-12-13T23:57:29.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.781+0000 7f6b066c9640 1 Processor -- start 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.781+0000 7f6b066c9640 1 -- start start 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b0015d330 0x7f6b000b8b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001601d0 con 0x7f6b000d2910 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b001604b0 con 0x7f6b0015d330 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b0014c050 con 0x7f6b000d3730 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57442/0 (socket says 172.21.15.124:57442) 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 -- 172.21.15.124:0/1121997639 learned_addr learned my addr 172.21.15.124:0/1121997639 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 -- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 msgr2=0x7f6b000b8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6aff7fe640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b04c66640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b0015d330 0x7f6b000b8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b8600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 -- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b0015d330 msgr2=0x7f6b000b8b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b0015d330 0x7f6b000b8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 -- 172.21.15.124:0/1121997639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af4009400 con 0x7f6b000d2910 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afffff640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6af000c4d0 tx=0x7f6af000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af00074e0 con 0x7f6b000d2910 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6af0007660 con 0x7f6b000d2910 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.782+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af8009b30 con 0x7f6b000d2910 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.783+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6af0004a20 con 0x7f6b000d2910 2023-12-13T23:57:29.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.783+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b0014c610 con 0x7f6b000d2910 2023-12-13T23:57:29.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.784+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b00069400 con 0x7f6b000d2910 2023-12-13T23:57:29.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.784+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6af0004b80 con 0x7f6b000d2910 2023-12-13T23:57:29.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.784+0000 7f6afcff9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 0x7f6aec06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:29.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.784+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6af0088b90 con 0x7f6b000d2910 2023-12-13T23:57:29.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.784+0000 7f6aff7fe640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 0x7f6aec06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:29.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.785+0000 7f6aff7fe640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 0x7f6aec06d680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6b000c9720 tx=0x7f6af4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:29.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.790+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6af0053950 con 0x7f6b000d2910 2023-12-13T23:57:29.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.955+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b000b5a20 con 0x7f6b000d2910 2023-12-13T23:57:29.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.956+0000 7f6afcff9640 1 -- 172.21.15.124:0/1121997639 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f6af0018020 con 0x7f6b000d2910 2023-12-13T23:57:29.956 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:29.956 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 msgr2=0x7f6aec06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 0x7f6aec06d680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6b000c9720 tx=0x7f6af4008040 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 msgr2=0x7f6b0014b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f6af000c4d0 tx=0x7f6af000c990 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 shutdown_connections 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6aec06b1c0 0x7f6aec06d680 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6b0015d330 0x7f6b000b8b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6b000d3730 0x7f6b000b8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 --2- 172.21.15.124:0/1121997639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b000d2910 0x7f6b0014b610 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 >> 172.21.15.124:0/1121997639 conn(0x7f6b00072020 msgr2=0x7f6b000d0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:29.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 shutdown_connections 2023-12-13T23:57:29.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:29.958+0000 7f6b066c9640 1 -- 172.21.15.124:0/1121997639 wait complete. 2023-12-13T23:57:29.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:29.969 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:30.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 Processor -- start 2023-12-13T23:57:30.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 -- start start 2023-12-13T23:57:30.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 0x7f084c0b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f084c0d2910 0x7f084c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.075 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084c0b2b10 con 0x7f084c0d3730 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c0d3bc0 con 0x7f084c0d2910 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.074+0000 7f085255a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084c0d1590 con 0x7f084c15d330 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f084c0d2910 0x7f084c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f084c0d2910 0x7f084c0d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57446/0 (socket says 172.21.15.124:57446) 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 -- 172.21.15.124:0/286858608 learned_addr learned my addr 172.21.15.124:0/286858608 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f0850af7640 1 --1- 172.21.15.124:0/286858608 >> v1:172.21.15.124:6789/0 conn(0x7f084c0d3730 0x7f084c0b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59260/0 (socket says 172.21.15.124:59260) 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084bfff640 1 --2- 172.21.15.124:0/286858608 >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 0x7f084c0b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 -- 172.21.15.124:0/286858608 >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 msgr2=0x7f084c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 --2- 172.21.15.124:0/286858608 >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 0x7f084c0b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 -- 172.21.15.124:0/286858608 >> v1:172.21.15.124:6789/0 conn(0x7f084c0d3730 legacy=0x7f084c0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084b7fe640 1 -- 172.21.15.124:0/286858608 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0d1870 con 0x7f084c0d2910 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084affd640 1 -- 172.21.15.124:0/286858608 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1311759383 0 0) 0x7f084c0b2b10 con 0x7f084c0d3730 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.075+0000 7f084bfff640 1 --2- 172.21.15.124:0/286858608 >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 0x7f084c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:30.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f084b7fe640 1 --2- 172.21.15.124:0/286858608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c0d31f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f084c0d0d10 tx=0x7f083402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e3d8d02bd22589 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f084affd640 1 -- 172.21.15.124:0/286858608 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834031870 con 0x7f084c0d2910 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f084affd640 1 -- 172.21.15.124:0/286858608 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08340319d0 con 0x7f084c0d2910 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f084affd640 1 -- 172.21.15.124:0/286858608 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0834031d60 con 0x7f084c0d2910 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 -- 172.21.15.124:0/286858608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 msgr2=0x7f084c0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 --2- 172.21.15.124:0/286858608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c0d31f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f084c0d0d10 tx=0x7f083402b910 comp rx=0 tx=0).stop 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 -- 172.21.15.124:0/286858608 shutdown_connections 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 --2- 172.21.15.124:0/286858608 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c0d31f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 --2- 172.21.15.124:0/286858608 >> v2:172.21.15.203:3300/0 conn(0x7f084c15d330 0x7f084c0b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 -- 172.21.15.124:0/286858608 >> 172.21.15.124:0/286858608 conn(0x7f084c072020 msgr2=0x7f084c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 -- 172.21.15.124:0/286858608 shutdown_connections 2023-12-13T23:57:30.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.076+0000 7f085255a640 1 -- 172.21.15.124:0/286858608 wait complete. 2023-12-13T23:57:30.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.077+0000 7f085255a640 1 Processor -- start 2023-12-13T23:57:30.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 -- start start 2023-12-13T23:57:30.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 0x7f084c15fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 0x7f084c0b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15e890 con 0x7f084c0d2910 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15dc80 con 0x7f084c15d330 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f085255a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084c15df60 con 0x7f084c0d3730 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f084bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f084bfff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57448/0 (socket says 172.21.15.124:57448) 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.078+0000 7f084bfff640 1 -- 172.21.15.124:0/834569710 learned_addr learned my addr 172.21.15.124:0/834569710 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f0850af7640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 0x7f084c0b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084b7fe640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 0x7f084c15fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 -- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 msgr2=0x7f084c15fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 0x7f084c15fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 -- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 msgr2=0x7f084c0b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 0x7f084c0b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 -- 172.21.15.124:0/834569710 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08340094d0 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084b7fe640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 0x7f084c15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f0850af7640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 0x7f084c0b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f084bfff640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f083c00c7f0 tx=0x7f083c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c007820 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083c007980 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c007cf0 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084c15e240 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.079+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084c160320 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.080+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084c069400 con 0x7f084c0d2910 2023-12-13T23:57:30.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.080+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f083c0113f0 con 0x7f084c0d2910 2023-12-13T23:57:30.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.081+0000 7f0848ff9640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 0x7f082c06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.081+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f083c089d40 con 0x7f084c0d2910 2023-12-13T23:57:30.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.081+0000 7f084b7fe640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 0x7f082c06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.082+0000 7f084b7fe640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 0x7f082c06d6b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0834036000 tx=0x7f083403c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.085+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f083c054a70 con 0x7f084c0d2910 2023-12-13T23:57:30.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.251+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f084c0b7410 con 0x7f084c0d2910 2023-12-13T23:57:30.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.252+0000 7f0848ff9640 1 -- 172.21.15.124:0/834569710 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f083c01f0a0 con 0x7f084c0d2910 2023-12-13T23:57:30.251 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:30.253 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 msgr2=0x7f082c06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.253 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 0x7f082c06d6b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0834036000 tx=0x7f083403c040 comp rx=0 tx=0).stop 2023-12-13T23:57:30.253 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 msgr2=0x7f084c158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f083c00c7f0 tx=0x7f083c00ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 shutdown_connections 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f082c06b1f0 0x7f082c06d6b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f084c15d330 0x7f084c0b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f084c0d3730 0x7f084c15fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 --2- 172.21.15.124:0/834569710 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f084c0d2910 0x7f084c158ae0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 >> 172.21.15.124:0/834569710 conn(0x7f084c072020 msgr2=0x7f084c072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 shutdown_connections 2023-12-13T23:57:30.254 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.254+0000 7f085255a640 1 -- 172.21.15.124:0/834569710 wait complete. 2023-12-13T23:57:30.264 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:30.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.367+0000 7fc615423640 1 Processor -- start 2023-12-13T23:57:30.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.367+0000 7fc615423640 1 -- start start 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc615423640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fc61015d1d0 0x7fc61006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc615423640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc61006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc615423640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6100adae0 con 0x7fc6100ad640 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc615423640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6100addc0 con 0x7fc61015d1d0 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc615423640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100ae0a0 con 0x7fc6100aeb80 2023-12-13T23:57:30.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60e7fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc6100ad640 0x7fc610160ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:59262/0 (socket says 172.21.15.124:59262) 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60e7fc640 1 -- 172.21.15.124:0/2840557406 learned_addr learned my addr 172.21.15.124:0/2840557406 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690831784 0 0) 0x7fc6100adae0 con 0x7fc6100ad640 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5f8003520 con 0x7fc6100ad640 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60effd640 1 --2- 172.21.15.124:0/2840557406 >> v2:172.21.15.203:3300/0 conn(0x7fc61015d1d0 0x7fc61006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc61006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5297446 0 0) 0x7fc5f8003520 con 0x7fc6100ad640 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 msgr2=0x7fc61006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc61006ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 >> v2:172.21.15.203:3300/0 conn(0x7fc61015d1d0 msgr2=0x7fc61006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 --2- 172.21.15.124:0/2840557406 >> v2:172.21.15.203:3300/0 conn(0x7fc61015d1d0 0x7fc61006b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6100ae380 con 0x7fc6100ad640 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.368+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fc604001670 con 0x7fc6100ad640 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 >> v1:172.21.15.124:6789/0 conn(0x7fc6100ad640 legacy=0x7fc610160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc5f8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f8003fa0 0x7fc5f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f8004d00 0x7fc5f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100addc0 con 0x7fc5f8004d00 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6100ae0a0 con 0x7fc5f8003fa0 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f800ba80 con 0x7fc6100aeb80 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f8004d00 0x7fc5f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60effd640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc5f8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60e7fc640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f8003fa0 0x7fc5f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 -- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 msgr2=0x7fc5f8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc6100aeb80 0x7fc5f8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 -- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f8003fa0 msgr2=0x7fc5f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f8003fa0 0x7fc5f80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60f7fe640 1 -- 172.21.15.124:0/2840557406 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8003520 con 0x7fc5f8004d00 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.369+0000 7fc60e7fc640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc5f8003fa0 0x7fc5f80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f8004d00 0x7fc5f8005180 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc60800ba30 tx=0x7fc5f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70a5a9e0d5d2f9d5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608032640 con 0x7fc5f8004d00 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6080327a0 con 0x7fc5f8004d00 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc60dffb640 1 -- 172.21.15.124:0/2840557406 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc608032b30 con 0x7fc5f8004d00 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 -- 172.21.15.124:0/2840557406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f8004d00 msgr2=0x7fc5f8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 --2- 172.21.15.124:0/2840557406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5f8004d00 0x7fc5f8005180 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fc60800ba30 tx=0x7fc5f80057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc60f7fe640 1 -- 172.21.15.124:0/2840557406 reap_dead start 2023-12-13T23:57:30.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 -- 172.21.15.124:0/2840557406 shutdown_connections 2023-12-13T23:57:30.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 -- 172.21.15.124:0/2840557406 >> 172.21.15.124:0/2840557406 conn(0x7fc610070e70 msgr2=0x7fc610071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 -- 172.21.15.124:0/2840557406 shutdown_connections 2023-12-13T23:57:30.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.370+0000 7fc615423640 1 -- 172.21.15.124:0/2840557406 wait complete. 2023-12-13T23:57:30.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.372+0000 7fc615423640 1 Processor -- start 2023-12-13T23:57:30.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.372+0000 7fc615423640 1 -- start start 2023-12-13T23:57:30.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.372+0000 7fc615423640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc61015d940 0x7fc610154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 0x7fc610154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc610155100 con 0x7fc6101542a0 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc610151a20 con 0x7fc61015ef50 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc61015deb0 con 0x7fc61015d940 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:57464/0 (socket says 172.21.15.124:57464) 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 -- 172.21.15.124:0/3141569230 learned_addr learned my addr 172.21.15.124:0/3141569230 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60e7fc640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc61015d940 0x7fc610154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 0x7fc610154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 -- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc61015d940 msgr2=0x7fc610154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc61015d940 0x7fc610154680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 -- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 msgr2=0x7fc610154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 0x7fc610154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 -- 172.21.15.124:0/3141569230 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc61015e190 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60f7fe640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 0x7fc610154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc60effd640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc6000049a0 tx=0x7fc600007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600015420 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc600015580 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc61015e470 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc600015930 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.373+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc61015ea30 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.374+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc610069400 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.375+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fc600015a90 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.375+0000 7fc5f77fe640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 0x7fc5fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.375+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc60008a3c0 con 0x7fc6101542a0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.375+0000 7fc60e7fc640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 0x7fc5fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.376+0000 7fc60e7fc640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 0x7fc5fc06d680 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc604004840 tx=0x7fc604007040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.379+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fc600055180 con 0x7fc6101542a0 2023-12-13T23:57:30.543 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.544+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc61006ab30 con 0x7fc6101542a0 2023-12-13T23:57:30.544 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.545+0000 7fc5f77fe640 1 -- 172.21.15.124:0/3141569230 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc600009cb0 con 0x7fc6101542a0 2023-12-13T23:57:30.544 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 msgr2=0x7fc5fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 0x7fc5fc06d680 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc604004840 tx=0x7fc604007040 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 msgr2=0x7fc610153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc6000049a0 tx=0x7fc600007700 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 shutdown_connections 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc5fc06b1c0 0x7fc5fc06d680 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fc61015ef50 0x7fc610154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fc61015d940 0x7fc610154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 --2- 172.21.15.124:0/3141569230 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6101542a0 0x7fc610153120 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 >> 172.21.15.124:0/3141569230 conn(0x7fc610070e70 msgr2=0x7fc61006b050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 shutdown_connections 2023-12-13T23:57:30.546 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.546+0000 7fc615423640 1 -- 172.21.15.124:0/3141569230 wait complete. 2023-12-13T23:57:30.557 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:30.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.660+0000 7fe60f7a9640 1 Processor -- start 2023-12-13T23:57:30.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.660+0000 7fe60f7a9640 1 -- start start 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60f7a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe60815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60f7a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe608071770 con 0x7fe6080b4370 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60f7a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6080ccb60 con 0x7fe6080ceb70 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60f7a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080ccdc0 con 0x7fe608144980 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60dd46640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe6080b4370 0x7fe608071c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44542/0 (socket says 172.21.15.124:44542) 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60dd46640 1 -- 172.21.15.124:0/66571530 learned_addr learned my addr 172.21.15.124:0/66571530 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:30.663 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.661+0000 7fe60d545640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe60815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407881177 0 0) 0x7fe608071770 con 0x7fe6080b4370 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f4003520 con 0x7fe6080b4370 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1763882766 0 0) 0x7fe5f4003520 con 0x7fe6080b4370 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 msgr2=0x7fe60815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe60815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 >> v1:172.21.15.203:6789/0 conn(0x7fe6080ceb70 legacy=0x7fe6080cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080cd0a0 con 0x7fe6080b4370 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fe600001670 con 0x7fe6080b4370 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 >> v1:172.21.15.124:6789/0 conn(0x7fe6080b4370 legacy=0x7fe608071c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe5f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5f4003ee0 0x7fe5f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4004d00 0x7fe5f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080ccdc0 con 0x7fe5f4003ee0 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f400b9e0 con 0x7fe5f4004d00 2023-12-13T23:57:30.664 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5f400bba0 con 0x7fe608144980 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5f4003ee0 0x7fe5f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60d545640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe5f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60dd46640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4004d00 0x7fe5f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 -- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 msgr2=0x7fe5f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608144980 0x7fe5f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 -- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4004d00 msgr2=0x7fe5f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4004d00 0x7fe5f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.662+0000 7fe60cd44640 1 -- 172.21.15.124:0/66571530 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f4003520 con 0x7fe5f4003ee0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.663+0000 7fe60cd44640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5f4003ee0 0x7fe5f40047c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80018e0 tx=0x7fe5f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c6e3f7b625c5414 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.663+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8032600 con 0x7fe5f4003ee0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.663+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f802e050 con 0x7fe5f4003ee0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.663+0000 7fe5fffff640 1 -- 172.21.15.124:0/66571530 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80329c0 con 0x7fe5f4003ee0 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 -- 172.21.15.124:0/66571530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5f4003ee0 msgr2=0x7fe5f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.665 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/66571530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe5f4003ee0 0x7fe5f40047c0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80018e0 tx=0x7fe5f40057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60dd46640 1 -- 172.21.15.124:0/66571530 reap_dead start 2023-12-13T23:57:30.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 -- 172.21.15.124:0/66571530 shutdown_connections 2023-12-13T23:57:30.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 -- 172.21.15.124:0/66571530 >> 172.21.15.124:0/66571530 conn(0x7fe60806a4d0 msgr2=0x7fe60806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 -- 172.21.15.124:0/66571530 shutdown_connections 2023-12-13T23:57:30.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.664+0000 7fe60f7a9640 1 -- 172.21.15.124:0/66571530 wait complete. 2023-12-13T23:57:30.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.665+0000 7fe60f7a9640 1 Processor -- start 2023-12-13T23:57:30.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.665+0000 7fe60f7a9640 1 -- start start 2023-12-13T23:57:30.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4003ee0 0x7fe608142b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608158220 0x7fe6081586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608158be0 con 0x7fe608154a50 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608135b30 con 0x7fe5f4003ee0 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe608154ed0 con 0x7fe608158220 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53118/0 (socket says 172.21.15.124:53118) 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 -- 172.21.15.124:0/1854256950 learned_addr learned my addr 172.21.15.124:0/1854256950 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60d545640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4003ee0 0x7fe608142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60dd46640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608158220 0x7fe6081586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 -- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608158220 msgr2=0x7fe6081586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608158220 0x7fe6081586a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 -- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4003ee0 msgr2=0x7fe608142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4003ee0 0x7fe608142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 -- 172.21.15.124:0/1854256950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608135db0 con 0x7fe608154a50 2023-12-13T23:57:30.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60cd44640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe5f802bc90 tx=0x7fe5f80012b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f80054e0 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f80035f0 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8003980 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.666+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe608136090 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.667+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6080acd30 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.668+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608069400 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.668+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe5f8039050 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.668+0000 7fe5fdffb640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 0x7fe5ec06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.669+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe5f80ab1b0 con 0x7fe608154a50 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.669+0000 7fe60d545640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 0x7fe5ec06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.669+0000 7fe60d545640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 0x7fe5ec06d6b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe604014510 tx=0x7fe60400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.673+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe5f8076050 con 0x7fe608154a50 2023-12-13T23:57:30.837 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.838+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe608155a80 con 0x7fe608154a50 2023-12-13T23:57:30.837 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.838+0000 7fe5fdffb640 1 -- 172.21.15.124:0/1854256950 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe5f8079670 con 0x7fe608154a50 2023-12-13T23:57:30.837 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 msgr2=0x7fe5ec06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 0x7fe5ec06d6b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe604014510 tx=0x7fe60400b040 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 msgr2=0x7fe6081355f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fe5f802bc90 tx=0x7fe5f80012b0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 shutdown_connections 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe5ec06b1f0 0x7fe5ec06d6b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe608158220 0x7fe6081586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe608154a50 0x7fe6081355f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 --2- 172.21.15.124:0/1854256950 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe5f4003ee0 0x7fe608142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 >> 172.21.15.124:0/1854256950 conn(0x7fe60806a4d0 msgr2=0x7fe608156580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 shutdown_connections 2023-12-13T23:57:30.840 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.840+0000 7fe60f7a9640 1 -- 172.21.15.124:0/1854256950 wait complete. 2023-12-13T23:57:30.850 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:57:30.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.951+0000 7fb63792c640 1 Processor -- start 2023-12-13T23:57:30.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.952+0000 7fb63792c640 1 -- start start 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.952+0000 7fb63792c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb630143490 0x7fb6300b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.952+0000 7fb63792c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63015e2a0 con 0x7fb63015e730 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.952+0000 7fb63792c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63015b240 con 0x7fb63015abf0 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.952+0000 7fb63792c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb63015b520 con 0x7fb630143490 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb6356c8640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb630143490 0x7fb6300b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb634ec7640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb63015e730 0x7fb63015a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44548/0 (socket says 172.21.15.124:44548) 2023-12-13T23:57:30.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb634ec7640 1 -- 172.21.15.124:0/352975546 learned_addr learned my addr 172.21.15.124:0/352975546 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:30.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb6356c8640 1 -- 172.21.15.124:0/352975546 >> v1:172.21.15.203:6789/0 conn(0x7fb63015abf0 legacy=0x7fb6300b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb6356c8640 1 -- 172.21.15.124:0/352975546 >> v1:172.21.15.124:6789/0 conn(0x7fb63015e730 legacy=0x7fb63015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.953+0000 7fb6356c8640 1 -- 172.21.15.124:0/352975546 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb63015dc50 con 0x7fb630143490 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb6356c8640 1 --2- 172.21.15.124:0/352975546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb630143490 0x7fb6300b0cf0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0097a0 tx=0x7fb62c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7773657a960358 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb627fff640 1 -- 172.21.15.124:0/352975546 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c030610 con 0x7fb630143490 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb627fff640 1 -- 172.21.15.124:0/352975546 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c030770 con 0x7fb630143490 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb627fff640 1 -- 172.21.15.124:0/352975546 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c030b00 con 0x7fb630143490 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 -- 172.21.15.124:0/352975546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb630143490 msgr2=0x7fb6300b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 --2- 172.21.15.124:0/352975546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb630143490 0x7fb6300b0cf0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fb62c0097a0 tx=0x7fb62c02b960 comp rx=0 tx=0).stop 2023-12-13T23:57:30.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 -- 172.21.15.124:0/352975546 shutdown_connections 2023-12-13T23:57:30.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 --2- 172.21.15.124:0/352975546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb630143490 0x7fb6300b0cf0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 -- 172.21.15.124:0/352975546 >> 172.21.15.124:0/352975546 conn(0x7fb63006a4d0 msgr2=0x7fb63006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:30.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 -- 172.21.15.124:0/352975546 shutdown_connections 2023-12-13T23:57:30.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.954+0000 7fb63792c640 1 -- 172.21.15.124:0/352975546 wait complete. 2023-12-13T23:57:30.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 Processor -- start 2023-12-13T23:57:30.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 -- start start 2023-12-13T23:57:30.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb630143490 0x7fb6301599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb63015abf0 0x7fb630140020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6301408b0 con 0x7fb63015e730 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb630140b90 con 0x7fb630143490 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb63792c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb630140e70 con 0x7fb63015abf0 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb635ec9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb635ec9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53136/0 (socket says 172.21.15.124:53136) 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.956+0000 7fb635ec9640 1 -- 172.21.15.124:0/1398198968 learned_addr learned my addr 172.21.15.124:0/1398198968 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb6356c8640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb630143490 0x7fb6301599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb634ec7640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb63015abf0 0x7fb630140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 -- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb63015abf0 msgr2=0x7fb630140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb63015abf0 0x7fb630140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 -- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb630143490 msgr2=0x7fb6301599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:30.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb630143490 0x7fb6301599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 -- 172.21.15.124:0/1398198968 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c009400 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb635ec9640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c7a0 tx=0x7fb62800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628003b70 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628003cf0 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb630141150 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628003ea0 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.957+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb630159f30 con 0x7fb63015e730 2023-12-13T23:57:30.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.958+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6300b4380 con 0x7fb63015e730 2023-12-13T23:57:30.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.959+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb62800aa90 con 0x7fb63015e730 2023-12-13T23:57:30.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.959+0000 7fb625ffb640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 0x7fb61406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:30.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.959+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb628089840 con 0x7fb63015e730 2023-12-13T23:57:30.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.959+0000 7fb6356c8640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 0x7fb61406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:30.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.960+0000 7fb6356c8640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 0x7fb61406d6b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb62c036000 tx=0x7fb62c008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:30.962 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:30.963+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb628054570 con 0x7fb63015e730 2023-12-13T23:57:31.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.145+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb630141670 con 0x7fb63015e730 2023-12-13T23:57:31.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.145+0000 7fb625ffb640 1 -- 172.21.15.124:0/1398198968 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb630141670 con 0x7fb63015e730 2023-12-13T23:57:31.144 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:57:31.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 msgr2=0x7fb61406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 0x7fb61406d6b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fb62c036000 tx=0x7fb62c008030 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 msgr2=0x7fb630141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c7a0 tx=0x7fb62800cc60 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 shutdown_connections 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb61406b1f0 0x7fb61406d6b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb63015e730 0x7fb630141950 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb63015abf0 0x7fb630140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 --2- 172.21.15.124:0/1398198968 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb630143490 0x7fb6301599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 >> 172.21.15.124:0/1398198968 conn(0x7fb63006a4d0 msgr2=0x7fb63006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 shutdown_connections 2023-12-13T23:57:31.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.147+0000 7fb63792c640 1 -- 172.21.15.124:0/1398198968 wait complete. 2023-12-13T23:57:31.157 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:31.157 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:31.172 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:31.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.312+0000 7fec41c17640 1 Processor -- start 2023-12-13T23:57:31.311 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.312+0000 7fec41c17640 1 -- start start 2023-12-13T23:57:31.312 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c143490 0x7fec3c0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec3c15e730 0x7fec3c0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 0x7fec3c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c15b520 con 0x7fec3c15e730 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec3c143870 con 0x7fec3c15a4e0 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec41c17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c15e2a0 con 0x7fec3c143490 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec3c15e730 0x7fec3c0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec3c15e730 0x7fec3c0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53146/0 (socket says 172.21.15.124:53146) 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 -- 172.21.15.124:0/1190320051 learned_addr learned my addr 172.21.15.124:0/1190320051 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3b7fe640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c143490 0x7fec3c0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3bfff640 1 --2- 172.21.15.124:0/1190320051 >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 0x7fec3c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 -- 172.21.15.124:0/1190320051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c143490 msgr2=0x7fec3c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c143490 0x7fec3c0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 -- 172.21.15.124:0/1190320051 >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 msgr2=0x7fec3c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:31.313 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 --2- 172.21.15.124:0/1190320051 >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 0x7fec3c15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3affd640 1 -- 172.21.15.124:0/1190320051 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3c15b240 con 0x7fec3c15e730 2023-12-13T23:57:31.314 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.313+0000 7fec3bfff640 1 --2- 172.21.15.124:0/1190320051 >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 0x7fec3c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec3affd640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15e730 0x7fec3c0b1230 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fec3c15eb90 tx=0x7fec2c02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a27ec01dd50913e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec3a7fc640 1 -- 172.21.15.124:0/1190320051 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c02ee90 con 0x7fec3c15e730 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec3a7fc640 1 -- 172.21.15.124:0/1190320051 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c036440 con 0x7fec3c15e730 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec3a7fc640 1 -- 172.21.15.124:0/1190320051 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0367d0 con 0x7fec3c15e730 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec41c17640 1 -- 172.21.15.124:0/1190320051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15e730 msgr2=0x7fec3c0b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.314+0000 7fec41c17640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15e730 0x7fec3c0b1230 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fec3c15eb90 tx=0x7fec2c02e0e0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 -- 172.21.15.124:0/1190320051 shutdown_connections 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15e730 0x7fec3c0b1230 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.315 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 --2- 172.21.15.124:0/1190320051 >> v2:172.21.15.203:3300/0 conn(0x7fec3c15a4e0 0x7fec3c15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 --2- 172.21.15.124:0/1190320051 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c143490 0x7fec3c0b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 -- 172.21.15.124:0/1190320051 >> 172.21.15.124:0/1190320051 conn(0x7fec3c06a4d0 msgr2=0x7fec3c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 -- 172.21.15.124:0/1190320051 shutdown_connections 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.315+0000 7fec41c17640 1 -- 172.21.15.124:0/1190320051 wait complete. 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.316+0000 7fec41c17640 1 Processor -- start 2023-12-13T23:57:31.316 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.316+0000 7fec41c17640 1 -- start start 2023-12-13T23:57:31.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec3c143490 0x7fec3c159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 0x7fec3c0ab090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.319 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c0abd50 con 0x7fec3c15a4e0 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c140af0 con 0x7fec3c143490 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec3c159d70 con 0x7fec3c15e730 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53162/0 (socket says 172.21.15.124:53162) 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 -- 172.21.15.124:0/1308420273 learned_addr learned my addr 172.21.15.124:0/1308420273 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3b7fe640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec3c143490 0x7fec3c159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3bfff640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 0x7fec3c0ab090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 -- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 msgr2=0x7fec3c0ab090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 0x7fec3c0ab090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 -- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec3c143490 msgr2=0x7fec3c159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec3c143490 0x7fec3c159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 -- 172.21.15.124:0/1308420273 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c00b770 con 0x7fec3c15a4e0 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3bfff640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 0x7fec3c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:31.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec3affd640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fec2c02e120 tx=0x7fec2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c034b80 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec2c034d00 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c041680 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3c140f10 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.317+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3c140150 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.318+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3c15e410 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.319+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fec2c03d040 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.319+0000 7fec23fff640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 0x7fec2806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.319+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec2c0ae9b0 con 0x7fec3c15a4e0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.319+0000 7fec3b7fe640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 0x7fec2806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.321 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.320+0000 7fec3b7fe640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 0x7fec2806d680 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec3000aac0 tx=0x7fec30008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.322 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.323+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fec2c0796e0 con 0x7fec3c15a4e0 2023-12-13T23:57:31.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.491+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec3c069400 con 0x7fec3c15a4e0 2023-12-13T23:57:31.491 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.492+0000 7fec23fff640 1 -- 172.21.15.124:0/1308420273 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7fec2c07cd00 con 0x7fec3c15a4e0 2023-12-13T23:57:31.491 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:31.492 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:31.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 msgr2=0x7fec2806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 0x7fec2806d680 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fec3000aac0 tx=0x7fec30008040 comp rx=0 tx=0).stop 2023-12-13T23:57:31.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 msgr2=0x7fec3c13f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fec2c02e120 tx=0x7fec2c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 shutdown_connections 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec2806b1c0 0x7fec2806d680 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec3c15e730 0x7fec3c0ab090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec3c15a4e0 0x7fec3c13f490 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 --2- 172.21.15.124:0/1308420273 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec3c143490 0x7fec3c159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 >> 172.21.15.124:0/1308420273 conn(0x7fec3c06a4d0 msgr2=0x7fec3c0718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 shutdown_connections 2023-12-13T23:57:31.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.494+0000 7fec41c17640 1 -- 172.21.15.124:0/1308420273 wait complete. 2023-12-13T23:57:31.504 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:31.505 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:31.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.608+0000 7f1cf5217640 1 Processor -- start 2023-12-13T23:57:31.607 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.608+0000 7f1cf5217640 1 -- start start 2023-12-13T23:57:31.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cf01415f0 0x7f1cf0141e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144a00 0x7f1cf006b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1cf0144440 0x7f1cf006ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.610 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf00b3b70 con 0x7f1cf01415f0 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cf00b2ef0 con 0x7f1cf0144440 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cf5217640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf00b31d0 con 0x7f1cf0144a00 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cf01415f0 0x7f1cf0141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1cf01415f0 0x7f1cf0141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53164/0 (socket says 172.21.15.124:53164) 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 -- 172.21.15.124:0/3418249271 learned_addr learned my addr 172.21.15.124:0/3418249271 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1cef59e640 1 --2- 172.21.15.124:0/3418249271 >> v2:172.21.15.203:3300/0 conn(0x7f1cf0144440 0x7f1cf006ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 -- 172.21.15.124:0/3418249271 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144a00 msgr2=0x7f1cf006b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/3418249271 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144a00 0x7f1cf006b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 -- 172.21.15.124:0/3418249271 >> v2:172.21.15.203:3300/0 conn(0x7f1cf0144440 msgr2=0x7f1cf006ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/3418249271 >> v2:172.21.15.203:3300/0 conn(0x7f1cf0144440 0x7f1cf006ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.609+0000 7f1ceed9d640 1 -- 172.21.15.124:0/3418249271 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf00b34b0 con 0x7f1cf01415f0 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/3418249271 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf0141e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00725a0 tx=0x7f1ce002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600d4ca11210344f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.611 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cee59c640 1 -- 172.21.15.124:0/3418249271 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0031870 con 0x7f1cf01415f0 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cee59c640 1 -- 172.21.15.124:0/3418249271 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce00319d0 con 0x7f1cf01415f0 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cee59c640 1 -- 172.21.15.124:0/3418249271 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0031d60 con 0x7f1cf01415f0 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 -- 172.21.15.124:0/3418249271 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 msgr2=0x7f1cf0141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 --2- 172.21.15.124:0/3418249271 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf0141e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00725a0 tx=0x7f1ce002b910 comp rx=0 tx=0).stop 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 -- 172.21.15.124:0/3418249271 shutdown_connections 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 --2- 172.21.15.124:0/3418249271 >> v2:172.21.15.203:3300/0 conn(0x7f1cf0144440 0x7f1cf006ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 --2- 172.21.15.124:0/3418249271 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144a00 0x7f1cf006b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.610+0000 7f1cf5217640 1 --2- 172.21.15.124:0/3418249271 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf0141e00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.611+0000 7f1cf5217640 1 -- 172.21.15.124:0/3418249271 >> 172.21.15.124:0/3418249271 conn(0x7f1cf0070e70 msgr2=0x7f1cf0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.611+0000 7f1cf5217640 1 -- 172.21.15.124:0/3418249271 shutdown_connections 2023-12-13T23:57:31.612 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.611+0000 7f1cf5217640 1 -- 172.21.15.124:0/3418249271 wait complete. 2023-12-13T23:57:31.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.612+0000 7f1cf5217640 1 Processor -- start 2023-12-13T23:57:31.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.612+0000 7f1cf5217640 1 -- start start 2023-12-13T23:57:31.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.612+0000 7f1cf5217640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.612+0000 7f1cf5217640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144440 0x7f1cf01f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.615 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cf0144a00 0x7f1cf0155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf0130020 con 0x7f1cf01415f0 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf0140410 con 0x7f1cf0144a00 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf01565d0 con 0x7f1cf0144440 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53174/0 (socket says 172.21.15.124:53174) 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 -- 172.21.15.124:0/2530633257 learned_addr learned my addr 172.21.15.124:0/2530633257 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cef59e640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cf0144a00 0x7f1cf0155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ce7fff640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144440 0x7f1cf01f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 -- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144440 msgr2=0x7f1cf01f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144440 0x7f1cf01f4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 -- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cf0144a00 msgr2=0x7f1cf0155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cf0144a00 0x7f1cf0155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 -- 172.21.15.124:0/2530633257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce00094d0 con 0x7f1cf01415f0 2023-12-13T23:57:31.616 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ceed9d640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0000c00 tx=0x7f1ce00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0031840 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce00443f0 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf01568b0 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce00447a0 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.613+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf0130550 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.614+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf0069400 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.615+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f1ce004d450 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.615+0000 7f1ce77fe640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 0x7f1cd006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.615+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ce002d040 con 0x7f1cf01415f0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.615+0000 7f1ce7fff640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 0x7f1cd006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.617 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.616+0000 7f1ce7fff640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 0x7f1cd006d680 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1cf015f0b0 tx=0x7f1cd8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.618 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.619+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f1ce0076f50 con 0x7f1cf01415f0 2023-12-13T23:57:31.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.783+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1cf00cc5c0 con 0x7f1cf01415f0 2023-12-13T23:57:31.783 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.784+0000 7f1ce77fe640 1 -- 172.21.15.124:0/2530633257 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1ce007a570 con 0x7f1cf01415f0 2023-12-13T23:57:31.783 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:31.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 msgr2=0x7f1cd006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 0x7f1cd006d680 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1cf015f0b0 tx=0x7f1cd8008040 comp rx=0 tx=0).stop 2023-12-13T23:57:31.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 msgr2=0x7f1cf01f4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ce0000c00 tx=0x7f1ce00046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 shutdown_connections 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f1cd006b1c0 0x7f1cd006d680 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1cf0144a00 0x7f1cf0155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1cf0144440 0x7f1cf01f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 --2- 172.21.15.124:0/2530633257 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1cf01415f0 0x7f1cf01f4560 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 >> 172.21.15.124:0/2530633257 conn(0x7f1cf0070e70 msgr2=0x7f1cf0071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 shutdown_connections 2023-12-13T23:57:31.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.786+0000 7f1cf5217640 1 -- 172.21.15.124:0/2530633257 wait complete. 2023-12-13T23:57:31.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:31.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.900+0000 7f283c6da640 1 Processor -- start 2023-12-13T23:57:31.900 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.900+0000 7f283c6da640 1 -- start start 2023-12-13T23:57:31.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283c6da640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f28340d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283c6da640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28340d3730 0x7f28340b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283c6da640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28340b2b10 con 0x7f283415d330 2023-12-13T23:57:31.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283c6da640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28340d3b50 con 0x7f28340d3730 2023-12-13T23:57:31.909 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283c6da640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28340d1590 con 0x7f28340d2910 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283ac77640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28340d3730 0x7f28340b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283ac77640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28340d3730 0x7f28340b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53190/0 (socket says 172.21.15.124:53190) 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283ac77640 1 -- 172.21.15.124:0/1999369910 learned_addr learned my addr 172.21.15.124:0/1999369910 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283a476640 1 --1- 172.21.15.124:0/1999369910 >> v1:172.21.15.124:6789/0 conn(0x7f283415d330 0x7f28340b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44564/0 (socket says 172.21.15.124:44564) 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f2839c75640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f28340d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283ac77640 1 -- 172.21.15.124:0/1999369910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 msgr2=0x7f28340d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.901+0000 7f283ac77640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f28340d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.910 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f283ac77640 1 -- 172.21.15.124:0/1999369910 >> v1:172.21.15.124:6789/0 conn(0x7f283415d330 legacy=0x7f28340b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f283ac77640 1 -- 172.21.15.124:0/1999369910 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28340d1870 con 0x7f28340d3730 2023-12-13T23:57:31.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f2839474640 1 -- 172.21.15.124:0/1999369910 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307556543 0 0) 0x7f28340b2b10 con 0x7f283415d330 2023-12-13T23:57:31.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f2839c75640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f28340d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:31.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f283ac77640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f28340b6390 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f282c009e50 tx=0x7f282c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f444c203cef28 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f2839474640 1 -- 172.21.15.124:0/1999369910 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c02ca70 con 0x7f28340d3730 2023-12-13T23:57:31.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f2839474640 1 -- 172.21.15.124:0/1999369910 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282c02cbd0 con 0x7f28340d3730 2023-12-13T23:57:31.912 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.902+0000 7f2839474640 1 -- 172.21.15.124:0/1999369910 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282c0343f0 con 0x7f28340d3730 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 -- 172.21.15.124:0/1999369910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 msgr2=0x7f28340b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f28340b6390 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f282c009e50 tx=0x7f282c02bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 -- 172.21.15.124:0/1999369910 shutdown_connections 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f28340d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 --2- 172.21.15.124:0/1999369910 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f28340b6390 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 -- 172.21.15.124:0/1999369910 >> 172.21.15.124:0/1999369910 conn(0x7f2834072020 msgr2=0x7f2834072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 -- 172.21.15.124:0/1999369910 shutdown_connections 2023-12-13T23:57:31.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.903+0000 7f283c6da640 1 -- 172.21.15.124:0/1999369910 wait complete. 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.904+0000 7f283c6da640 1 Processor -- start 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.904+0000 7f283c6da640 1 -- start start 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f2834152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f283415d330 0x7f28340c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28341f7210 con 0x7f28340d3730 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28340cadc0 con 0x7f283415d330 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283c6da640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28340ca4d0 con 0x7f28340d2910 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53204/0 (socket says 172.21.15.124:53204) 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 -- 172.21.15.124:0/1596369192 learned_addr learned my addr 172.21.15.124:0/1596369192 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283ac77640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f283415d330 0x7f28340c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f283a476640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f2834152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 -- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 msgr2=0x7f2834152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f2834152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 -- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f283415d330 msgr2=0x7f28340c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f283415d330 0x7f28340c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 -- 172.21.15.124:0/1596369192 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282c009b30 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.905+0000 7f2839c75640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f282400c7f0 tx=0x7f282400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824007820 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2824007980 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28340ca770 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2824007cf0 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2834152570 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.906+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834069400 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.907+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f28240113f0 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.907+0000 7f282affd640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 0x7f281806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.907+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2824089d30 con 0x7f28340d3730 2023-12-13T23:57:31.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.908+0000 7f283a476640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 0x7f281806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:31.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.908+0000 7f283a476640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 0x7f281806d680 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2830000f60 tx=0x7f2830007b50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:31.916 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:31.912+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2824054a60 con 0x7f28340d3730 2023-12-13T23:57:32.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.079+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f28340b2050 con 0x7f28340d3730 2023-12-13T23:57:32.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.080+0000 7f282affd640 1 -- 172.21.15.124:0/1596369192 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f282401f020 con 0x7f28340d3730 2023-12-13T23:57:32.079 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:32.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 msgr2=0x7f281806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 0x7f281806d680 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2830000f60 tx=0x7f2830007b50 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 msgr2=0x7f2834152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f282400c7f0 tx=0x7f282400ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 shutdown_connections 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f281806b1c0 0x7f281806d680 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f283415d330 0x7f28340c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28340d3730 0x7f2834152800 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 --2- 172.21.15.124:0/1596369192 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f28340d2910 0x7f2834152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 >> 172.21.15.124:0/1596369192 conn(0x7f2834072020 msgr2=0x7f2834072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 shutdown_connections 2023-12-13T23:57:32.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.082+0000 7f283c6da640 1 -- 172.21.15.124:0/1596369192 wait complete. 2023-12-13T23:57:32.092 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:32.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.195+0000 7fd8ed27c640 1 Processor -- start 2023-12-13T23:57:32.195 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.196+0000 7fd8ed27c640 1 -- start start 2023-12-13T23:57:32.198 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.196+0000 7fd8ed27c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd8e8160fb0 0x7fd8e8161440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.196+0000 7fd8ed27c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e8161d20 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.196+0000 7fd8ed27c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8e815e760 con 0x7fd8e806ea50 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.196+0000 7fd8ed27c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8e815d9a0 con 0x7fd8e8160fb0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e77fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd8e80d0ed0 0x7fd8e80d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44574/0 (socket says 172.21.15.124:44574) 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e77fe640 1 -- 172.21.15.124:0/2622718140 learned_addr learned my addr 172.21.15.124:0/2622718140 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/2622718140 >> v2:172.21.15.203:3300/0 conn(0x7fd8e8160fb0 0x7fd8e8161440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243475927 0 0) 0x7fd8e8161d20 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d0003520 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 744145576 0 0) 0x7fd8d0003520 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 >> v2:172.21.15.203:3300/0 conn(0x7fd8e8160fb0 msgr2=0x7fd8e8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 --2- 172.21.15.124:0/2622718140 >> v2:172.21.15.203:3300/0 conn(0x7fd8e8160fb0 0x7fd8e8161440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 >> v1:172.21.15.203:6789/0 conn(0x7fd8e806ea50 legacy=0x7fd8e806ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.199 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e815dc80 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fd8e00019f0 con 0x7fd8e80d0ed0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 >> v1:172.21.15.124:6789/0 conn(0x7fd8e80d0ed0 legacy=0x7fd8e80d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d0003970 0x7fd8d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8d0005180 0x7fd8d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815d9a0 con 0x7fd8d00047c0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d000c120 con 0x7fd8d0003970 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d000c2e0 con 0x7fd8d0005180 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e6ffd640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d0003970 0x7fd8d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.197+0000 7fd8e77fe640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8d0005180 0x7fd8d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 -- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8d0005180 msgr2=0x7fd8d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8d0005180 0x7fd8d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 -- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d0003970 msgr2=0x7fd8d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:32.200 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8d0003970 0x7fd8d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 -- 172.21.15.124:0/2622718140 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d0003520 con 0x7fd8d00047c0 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e77fe640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8d0005180 0x7fd8d000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e77fe640 1 -- 172.21.15.124:0/2622718140 reap_dead start 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8d0004c40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009ac0 tx=0x7fd8d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20f052507fc1f1e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc02c6b0 con 0x7fd8d00047c0 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8dc02c810 con 0x7fd8d00047c0 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.198+0000 7fd8e5ffb640 1 -- 172.21.15.124:0/2622718140 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8dc02cba0 con 0x7fd8d00047c0 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 -- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 msgr2=0x7fd8d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8d0004c40 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc009ac0 tx=0x7fd8d00057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 -- 172.21.15.124:0/2622718140 shutdown_connections 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/2622718140 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8d0004c40 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 -- 172.21.15.124:0/2622718140 >> 172.21.15.124:0/2622718140 conn(0x7fd8e806a6b0 msgr2=0x7fd8e806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 -- 172.21.15.124:0/2622718140 shutdown_connections 2023-12-13T23:57:32.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.199+0000 7fd8ed27c640 1 -- 172.21.15.124:0/2622718140 wait complete. 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.200+0000 7fd8ed27c640 1 Processor -- start 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.200+0000 7fd8ed27c640 1 -- start start 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8e80cbbe0 0x7fd8e80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8e8159340 0x7fd8e81597c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e80cc5a0 con 0x7fd8d00047c0 2023-12-13T23:57:32.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e8157120 con 0x7fd8e80cbbe0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8ed27c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e815a000 con 0x7fd8e8159340 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53220/0 (socket says 172.21.15.124:53220) 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 -- 172.21.15.124:0/3581629784 learned_addr learned my addr 172.21.15.124:0/3581629784 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e77fe640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8e8159340 0x7fd8e81597c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8e80cbbe0 0x7fd8e80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 -- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8e8159340 msgr2=0x7fd8e81597c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8e8159340 0x7fd8e81597c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 -- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8e80cbbe0 msgr2=0x7fd8e80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8e80cbbe0 0x7fd8e80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.201+0000 7fd8e6ffd640 1 -- 172.21.15.124:0/3581629784 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e815a2e0 con 0x7fd8d00047c0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8e6ffd640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800e4d0 tx=0x7fd8d800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d80053a0 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8d80035f0 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e815a5c0 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8d8003980 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.202+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e815ab80 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.203+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e815b290 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.203+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd8d8003ae0 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.203+0000 7fd8cf7fe640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 0x7fd8d406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.204+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd8d80897f0 con 0x7fd8d00047c0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.204+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 0x7fd8d406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.204+0000 7fd8e67fc640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 0x7fd8d406d6b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc036000 tx=0x7fd8dc030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.208+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd8d8054520 con 0x7fd8d00047c0 2023-12-13T23:57:32.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.375+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd8e8069400 con 0x7fd8d00047c0 2023-12-13T23:57:32.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.375+0000 7fd8cf7fe640 1 -- 172.21.15.124:0/3581629784 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd8d8057b40 con 0x7fd8d00047c0 2023-12-13T23:57:32.375 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:32.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 msgr2=0x7fd8d406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 0x7fd8d406d6b0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd8dc036000 tx=0x7fd8dc030000 comp rx=0 tx=0).stop 2023-12-13T23:57:32.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 msgr2=0x7fd8e8156be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fd8d800e4d0 tx=0x7fd8d800e990 comp rx=0 tx=0).stop 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 shutdown_connections 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd8d406b1f0 0x7fd8d406d6b0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd8e80cbbe0 0x7fd8e80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd8e8159340 0x7fd8e81597c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 --2- 172.21.15.124:0/3581629784 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd8d00047c0 0x7fd8e8156be0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 >> 172.21.15.124:0/3581629784 conn(0x7fd8e806a6b0 msgr2=0x7fd8e815fc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 shutdown_connections 2023-12-13T23:57:32.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.377+0000 7fd8ed27c640 1 -- 172.21.15.124:0/3581629784 wait complete. 2023-12-13T23:57:32.387 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:57:32.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 Processor -- start 2023-12-13T23:57:32.488 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 -- start start 2023-12-13T23:57:32.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 0x7fca100b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca100b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.489 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca100b2b10 con 0x7fca100d2910 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca100d1590 con 0x7fca1015d330 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.489+0000 7fca15831640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100d1870 con 0x7fca100d3900 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0effd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 0x7fca100b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0f7fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca100b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0e7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fca100d2910 0x7fca100d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:47448/0 (socket says 172.21.15.124:47448) 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0e7fc640 1 -- 172.21.15.124:0/1899207629 learned_addr learned my addr 172.21.15.124:0/1899207629 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0dffb640 1 -- 172.21.15.124:0/1899207629 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288849036 0 0) 0x7fca100b2b10 con 0x7fca100d2910 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0dffb640 1 -- 172.21.15.124:0/1899207629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f8003520 con 0x7fca100d2910 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0f7fe640 1 -- 172.21.15.124:0/1899207629 >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 msgr2=0x7fca100b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0f7fe640 1 --2- 172.21.15.124:0/1899207629 >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 0x7fca100b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0f7fe640 1 -- 172.21.15.124:0/1899207629 >> v1:172.21.15.203:6789/0 conn(0x7fca100d2910 legacy=0x7fca100d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0f7fe640 1 -- 172.21.15.124:0/1899207629 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca100d1b50 con 0x7fca100d3900 2023-12-13T23:57:32.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.490+0000 7fca0effd640 1 --2- 172.21.15.124:0/1899207629 >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 0x7fca100b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.491+0000 7fca0f7fe640 1 --2- 172.21.15.124:0/1899207629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca100b6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca080097a0 tx=0x7fca0802b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac3b2fee92a5a44c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.491+0000 7fca0dffb640 1 -- 172.21.15.124:0/1899207629 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca0802e070 con 0x7fca100d3900 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.491+0000 7fca0dffb640 1 -- 172.21.15.124:0/1899207629 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca080315a0 con 0x7fca100d3900 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.491+0000 7fca0dffb640 1 -- 172.21.15.124:0/1899207629 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08031930 con 0x7fca100d3900 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 -- 172.21.15.124:0/1899207629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 msgr2=0x7fca100b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 --2- 172.21.15.124:0/1899207629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca100b6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fca080097a0 tx=0x7fca0802b840 comp rx=0 tx=0).stop 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 -- 172.21.15.124:0/1899207629 shutdown_connections 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 --2- 172.21.15.124:0/1899207629 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca100b6390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 --2- 172.21.15.124:0/1899207629 >> v2:172.21.15.203:3300/0 conn(0x7fca1015d330 0x7fca100b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 -- 172.21.15.124:0/1899207629 >> 172.21.15.124:0/1899207629 conn(0x7fca10072020 msgr2=0x7fca10072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 -- 172.21.15.124:0/1899207629 shutdown_connections 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.492+0000 7fca15831640 1 -- 172.21.15.124:0/1899207629 wait complete. 2023-12-13T23:57:32.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.493+0000 7fca15831640 1 Processor -- start 2023-12-13T23:57:32.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.493+0000 7fca15831640 1 -- start start 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca100d2910 0x7fca100c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca10152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca101f71e0 con 0x7fca1015d330 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10152db0 con 0x7fca100d2910 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca101f6310 con 0x7fca100d3900 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53230/0 (socket says 172.21.15.124:53230) 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 -- 172.21.15.124:0/2344472973 learned_addr learned my addr 172.21.15.124:0/2344472973 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0e7fc640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca10152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0effd640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca100d2910 0x7fca100c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 -- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 msgr2=0x7fca10152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca10152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 -- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca100d2910 msgr2=0x7fca100c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca100d2910 0x7fca100c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 -- 172.21.15.124:0/2344472973 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca00009a70 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0e7fc640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca10152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca0f7fe640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fca080098c0 tx=0x7fca080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08032840 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca080329a0 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca08009400 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.494+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca08032d50 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.495+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca101f6840 con 0x7fca1015d330 2023-12-13T23:57:32.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.496+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca100d23b0 con 0x7fca1015d330 2023-12-13T23:57:32.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.496+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fca080453f0 con 0x7fca1015d330 2023-12-13T23:57:32.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.496+0000 7fc9f7fff640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 0x7fc9fc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:32.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.497+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca080aba00 con 0x7fca1015d330 2023-12-13T23:57:32.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.497+0000 7fca0effd640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 0x7fc9fc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:32.497 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.497+0000 7fca0effd640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 0x7fc9fc06d680 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca00007600 tx=0x7fca0000b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:32.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.501+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fca080768e0 con 0x7fca1015d330 2023-12-13T23:57:32.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.678+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca10069400 con 0x7fca1015d330 2023-12-13T23:57:32.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.679+0000 7fc9f7fff640 1 -- 172.21.15.124:0/2344472973 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fca08079f00 con 0x7fca1015d330 2023-12-13T23:57:32.678 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:57:32.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 msgr2=0x7fc9fc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.680 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 0x7fc9fc06d680 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca00007600 tx=0x7fca0000b040 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 msgr2=0x7fca10152870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fca080098c0 tx=0x7fca080046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 shutdown_connections 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fc9fc06b1c0 0x7fc9fc06d680 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fca1015d330 0x7fca10152870 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fca100d3900 0x7fca10152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 --2- 172.21.15.124:0/2344472973 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fca100d2910 0x7fca100c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 >> 172.21.15.124:0/2344472973 conn(0x7fca10072020 msgr2=0x7fca100d0050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 shutdown_connections 2023-12-13T23:57:32.681 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:32.681+0000 7fca15831640 1 -- 172.21.15.124:0/2344472973 wait complete. 2023-12-13T23:57:32.691 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:57:32.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:32.691 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:32.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[109859]: fuse finished with error 0 and tester_r 0 2023-12-13T23:57:32.718 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:57:38.820 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:38.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:38.820 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:57:38.839 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-12-13T23:57:38.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.986+0000 7f50e62bb640 1 Processor -- start 2023-12-13T23:57:38.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.986+0000 7f50e62bb640 1 -- start start 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50e62bb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50e00b1400 0x7f50e015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50e62bb640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e015e2a0 con 0x7f50e0143490 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50e62bb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e015b240 con 0x7f50e015e730 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50e62bb640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e015b520 con 0x7f50e00b1400 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50df7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50e00b1400 0x7f50e015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50df7fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f50e00b1400 0x7f50e015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53246/0 (socket says 172.21.15.124:53246) 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50dffff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f50e0143490 0x7f50e00b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44584/0 (socket says 172.21.15.124:44584) 2023-12-13T23:57:38.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.987+0000 7f50df7fe640 1 -- 172.21.15.124:0/2876645894 learned_addr learned my addr 172.21.15.124:0/2876645894 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:38.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50df7fe640 1 -- 172.21.15.124:0/2876645894 >> v1:172.21.15.203:6789/0 conn(0x7f50e015e730 legacy=0x7f50e015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:38.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50df7fe640 1 -- 172.21.15.124:0/2876645894 >> v1:172.21.15.124:6789/0 conn(0x7f50e0143490 legacy=0x7f50e00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:38.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50df7fe640 1 -- 172.21.15.124:0/2876645894 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e015dc50 con 0x7f50e00b1400 2023-12-13T23:57:38.988 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50deffd640 1 -- 172.21.15.124:0/2876645894 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326872108 0 0) 0x7f50e015e2a0 con 0x7f50e0143490 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50df7fe640 1 --2- 172.21.15.124:0/2876645894 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e00b1400 0x7f50e015a4e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009870 tx=0x7f50d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dfe027d66c523d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50deffd640 1 -- 172.21.15.124:0/2876645894 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4030860 con 0x7f50e00b1400 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.988+0000 7f50deffd640 1 -- 172.21.15.124:0/2876645894 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d40309c0 con 0x7f50e00b1400 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50deffd640 1 -- 172.21.15.124:0/2876645894 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4030d50 con 0x7f50e00b1400 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 -- 172.21.15.124:0/2876645894 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e00b1400 msgr2=0x7f50e015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2876645894 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e00b1400 0x7f50e015a4e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f50d4009870 tx=0x7f50d402b9a0 comp rx=0 tx=0).stop 2023-12-13T23:57:38.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 -- 172.21.15.124:0/2876645894 shutdown_connections 2023-12-13T23:57:38.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2876645894 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e00b1400 0x7f50e015a4e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:38.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 -- 172.21.15.124:0/2876645894 >> 172.21.15.124:0/2876645894 conn(0x7f50e006a4d0 msgr2=0x7f50e006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:38.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 -- 172.21.15.124:0/2876645894 shutdown_connections 2023-12-13T23:57:38.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.989+0000 7f50e62bb640 1 -- 172.21.15.124:0/2876645894 wait complete. 2023-12-13T23:57:38.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.990+0000 7f50e62bb640 1 Processor -- start 2023-12-13T23:57:38.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 -- start start 2023-12-13T23:57:38.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e00b1400 0x7f50e015d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:38.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 0x7f50e006c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e0141ef0 con 0x7f50e015e730 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e0163a80 con 0x7f50e0143490 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e62bb640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e0142190 con 0x7f50e00b1400 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e4858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e4858640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:53250/0 (socket says 172.21.15.124:53250) 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50e4858640 1 -- 172.21.15.124:0/2668913225 learned_addr learned my addr 172.21.15.124:0/2668913225 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50dffff640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e00b1400 0x7f50e015d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.991+0000 7f50df7fe640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 0x7f50e006c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 -- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e00b1400 msgr2=0x7f50e015d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e00b1400 0x7f50e015d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:38.993 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 -- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 msgr2=0x7f50e006c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 0x7f50e006c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 -- 172.21.15.124:0/2668913225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40094d0 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50df7fe640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 0x7f50e006c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e4858640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f50d800b970 tx=0x7f50d800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d800cb40 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d800cca0 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e0159a20 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d8014680 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.992+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e015bc20 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.993+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e0163c40 con 0x7f50e015e730 2023-12-13T23:57:38.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.993+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f50d801d3b0 con 0x7f50e015e730 2023-12-13T23:57:38.995 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.994+0000 7f50dcff9640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:38.995 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.994+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50d80899c0 con 0x7f50e015e730 2023-12-13T23:57:38.995 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.994+0000 7f50dffff640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:38.995 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.994+0000 7f50dffff640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f50d0000f60 tx=0x7f50d0007d50 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:38.997 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:38.998+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f50d8054710 con 0x7f50e015e730 2023-12-13T23:57:39.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.202+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f50e0069400 con 0x7f50e015e730 2023-12-13T23:57:39.201 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.202+0000 7f50dcff9640 1 -- 172.21.15.124:0/2668913225 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v22) v1 ==== 53+0+73 (secure 0 0 0) 0x7f50d8019020 con 0x7f50e015e730 2023-12-13T23:57:39.201 INFO:teuthology.orchestra.run.smithi124.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 msgr2=0x7f50cc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f50d0000f60 tx=0x7f50d0007d50 comp rx=0 tx=0).stop 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 msgr2=0x7f50e01f3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f50d800b970 tx=0x7f50d800be30 comp rx=0 tx=0).stop 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 shutdown_connections 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f50cc06b1c0 0x7f50cc06d680 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f50e0143490 0x7f50e006c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50e015e730 0x7f50e01f3420 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:39.204 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 --2- 172.21.15.124:0/2668913225 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50e00b1400 0x7f50e015d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:39.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 >> 172.21.15.124:0/2668913225 conn(0x7f50e006a4d0 msgr2=0x7f50e013fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:39.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 shutdown_connections 2023-12-13T23:57:39.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:39.204+0000 7f50e62bb640 1 -- 172.21.15.124:0/2668913225 wait complete. 2023-12-13T23:57:39.214 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-12-13T23:57:39.214 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-12-13T23:57:39.215 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:39.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.215 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:39.231 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42547sec preferred_lft 42547sec 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-12-13T23:57:39.232 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::8896:c7ff:fe6e:3d67/64 scope link 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 2a:a2:ad:78:43:e8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::28a2:adff:fe78:43e8/64 scope link 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether de:7b:05:65:3c:b8 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::dc7b:5ff:fe65:3cb8/64 scope link 2023-12-13T23:57:39.233 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-12-13T23:57:39.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.234 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-12-13T23:57:39.297 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-12-13T23:57:39.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.298 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-12-13T23:57:39.354 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-12-13T23:57:39.354 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-12-13T23:57:39.354 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-12-13T23:57:39.355 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-12-13T23:57:39.355 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-12-13T23:57:39.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-12-13T23:57:39.355 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-12-13T23:57:39.450 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.450 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:57:39.465 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:57:39.465 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:57:39.465 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:39.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.466 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-12-13T23:57:39.535 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2023-12-13T23:57:39.535 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:39.535 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:39.592 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-12-13T23:57:39.592 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-12-13T23:57:39.593 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:39.648 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:39.715 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:39.772 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:39.773 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:39.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.775 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:39.826 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-12-13T23:57:39.827 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2023-12-13T23:57:39.869 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2023-12-13T23:57:39.895 DEBUG:teuthology.orchestra.run.smithi124:> cat /proc/self/mounts | awk '{print $2}' 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/proc 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/dev 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/security 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/shm 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/pts 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/run 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/cgroup 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/pstore 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/bpf 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/ 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/selinux 2023-12-13T23:57:39.952 INFO:teuthology.orchestra.run.smithi124.stdout:/proc/sys/fs/binfmt_misc 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/hugepages 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/dev/mqueue 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/debug 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/tracing 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/fs/fuse/connections 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/sys/kernel/config 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-sysctl.service 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/run/credentials/systemd-tmpfiles-setup.service 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/nfs/rpc_pipefs 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/run/user/1000 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph 2023-12-13T23:57:39.953 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-0 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-1 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-2 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/var/lib/ceph/osd/ceph-3 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run.smithi124.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:39.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:39.954 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2023-12-13T23:57:39.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2023-12-13T23:57:39.954+0000 7fb9de84a480 -1 init, newargv = 0x5625d4a24b10 newargc=15 2023-12-13T23:57:39.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[110821]: starting ceph client 2023-12-13T23:57:39.958 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[110821]: starting fuse 2023-12-13T23:57:39.970 INFO:teuthology.orchestra.run.smithi124.stdout:41 2023-12-13T23:57:39.970 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2023-12-13T23:57:39.970 DEBUG:teuthology.orchestra.run.smithi124:> sudo stdin-killer -- python3 -c ' 2023-12-13T23:57:39.970 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2023-12-13T23:57:39.970 DEBUG:teuthology.orchestra.run.smithi124:> import re 2023-12-13T23:57:39.970 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> def _find_admin_socket(client_name): 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-12-13T23:57:39.971 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> contents = proc_f.read() 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> if mountpoint in contents: 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> return f 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> print(_find_admin_socket("client.0")) 2023-12-13T23:57:39.972 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:57:40.090 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.110821.asok 2023-12-13T23:57:40.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2023-12-13T23:57:40.100 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.110821.asok 2023-12-13T23:57:40.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:40.101 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.110821.asok status 2023-12-13T23:57:40.236 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "021ac1670a21c19759080c6da90baaeb42e7d175", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 19.0.0-157-g021ac167 (021ac1670a21c19759080c6da90baaeb42e7d175) squid (dev)", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "110821", 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6225, 2023-12-13T23:57:40.237 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6225 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2816567271 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: } 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "192.168.0.1:0", 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2816567271 2023-12-13T23:57:40.238 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6225 192.168.0.1:0/2816567271", 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "192.168.0.1:0/2816567271", 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 22, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 50, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "blocklisted": false, 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2023-12-13T23:57:40.239 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:40.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:40.245 DEBUG:teuthology.orchestra.run.smithi124:> stat --file-system '--printf=%T 2023-12-13T23:57:40.245 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:40.260 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2023-12-13T23:57:40.260 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:40.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:40.261 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:40.327 DEBUG:teuthology.orchestra.run.smithi124:> stdin-killer -- python3 -c ' 2023-12-13T23:57:40.327 DEBUG:teuthology.orchestra.run.smithi124:> import os 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> import errno 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> try: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> if True: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> ret = os.read(fd, 1024) 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> else: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> os.write(fd, b'"'"'content'"'"') 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> except OSError as e: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> if e.errno != errno.EPERM: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> raise 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> else: 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("client does not check permission of data pool") 2023-12-13T23:57:40.328 DEBUG:teuthology.orchestra.run.smithi124:> ' 2023-12-13T23:57:40.434 INFO:teuthology.orchestra.run.smithi124.stderr:Traceback (most recent call last): 2023-12-13T23:57:40.435 INFO:teuthology.orchestra.run.smithi124.stderr: File "", line 15, in 2023-12-13T23:57:40.435 INFO:teuthology.orchestra.run.smithi124.stderr:RuntimeError: client does not check permission of data pool 2023-12-13T23:57:40.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2023-12-13T23:57:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:40.443 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2023-12-13T23:57:40.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.591+0000 7f2a7774c640 1 Processor -- start 2023-12-13T23:57:40.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.591+0000 7f2a7774c640 1 -- start start 2023-12-13T23:57:40.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a7774c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 0x7f2a700ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a7774c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a7774c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a70141d10 con 0x7f2a70144600 2023-12-13T23:57:40.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a7774c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7015fdc0 con 0x7f2a7015f8c0 2023-12-13T23:57:40.591 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a7774c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700cd320 con 0x7f2a70141930 2023-12-13T23:57:40.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a74ce7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a754e8640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 0x7f2a700ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a74ce7640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:50342/0 (socket says 172.21.15.124:50342) 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a74ce7640 1 -- 172.21.15.124:0/854284783 learned_addr learned my addr 172.21.15.124:0/854284783 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a754e8640 1 -- 172.21.15.124:0/854284783 >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 msgr2=0x7f2a700b0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a754e8640 1 --2- 172.21.15.124:0/854284783 >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a754e8640 1 -- 172.21.15.124:0/854284783 >> v1:172.21.15.203:6789/0 conn(0x7f2a70144600 legacy=0x7f2a70141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a754e8640 1 -- 172.21.15.124:0/854284783 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a700cd600 con 0x7f2a70141930 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.592+0000 7f2a74ce7640 1 --2- 172.21.15.124:0/854284783 >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.593+0000 7f2a754e8640 1 --2- 172.21.15.124:0/854284783 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 0x7f2a700ce0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0097a0 tx=0x7f2a6c02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d915b815dd7de786 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.593+0000 7f2a67fff640 1 -- 172.21.15.124:0/854284783 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c02e070 con 0x7f2a70141930 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.593+0000 7f2a67fff640 1 -- 172.21.15.124:0/854284783 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6c0315a0 con 0x7f2a70141930 2023-12-13T23:57:40.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.593+0000 7f2a67fff640 1 -- 172.21.15.124:0/854284783 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6c031930 con 0x7f2a70141930 2023-12-13T23:57:40.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 -- 172.21.15.124:0/854284783 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 msgr2=0x7f2a700ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.594 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 --2- 172.21.15.124:0/854284783 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 0x7f2a700ce0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c0097a0 tx=0x7f2a6c02b840 comp rx=0 tx=0).stop 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 -- 172.21.15.124:0/854284783 shutdown_connections 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 --2- 172.21.15.124:0/854284783 >> v2:172.21.15.203:3300/0 conn(0x7f2a7015f8c0 0x7f2a700b0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 --2- 172.21.15.124:0/854284783 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70141930 0x7f2a700ce0e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 -- 172.21.15.124:0/854284783 >> 172.21.15.124:0/854284783 conn(0x7f2a7006d9c0 msgr2=0x7f2a7006a4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 -- 172.21.15.124:0/854284783 shutdown_connections 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.594+0000 7f2a7774c640 1 -- 172.21.15.124:0/854284783 wait complete. 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.595+0000 7f2a7774c640 1 Processor -- start 2023-12-13T23:57:40.595 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.595+0000 7f2a7774c640 1 -- start start 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 0x7f2a701f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70144600 0x7f2a700cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700ab3f0 con 0x7f2a7015f8c0 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a700c4410 con 0x7f2a70141930 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a7774c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7014fa60 con 0x7f2a70144600 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41468/0 (socket says 172.21.15.124:41468) 2023-12-13T23:57:40.597 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 -- 172.21.15.124:0/4138206224 learned_addr learned my addr 172.21.15.124:0/4138206224 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a754e8640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 0x7f2a701f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a74ce7640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70144600 0x7f2a700cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 -- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70144600 msgr2=0x7f2a700cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70144600 0x7f2a700cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 -- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 msgr2=0x7f2a701f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 0x7f2a701f5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 -- 172.21.15.124:0/4138206224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a600094d0 con 0x7f2a7015f8c0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a754e8640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 0x7f2a701f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.596+0000 7f2a75ce9640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c6e0 tx=0x7f2a6800cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.597+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68003b70 con 0x7f2a7015f8c0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.597+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68003cf0 con 0x7f2a7015f8c0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.597+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c009400 con 0x7f2a7015f8c0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.597+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005310 con 0x7f2a7015f8c0 2023-12-13T23:57:40.598 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.597+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7014fd00 con 0x7f2a7015f8c0 2023-12-13T23:57:40.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.598+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7014ff20 con 0x7f2a7015f8c0 2023-12-13T23:57:40.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.598+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2a68005470 con 0x7f2a7015f8c0 2023-12-13T23:57:40.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.599+0000 7f2a65ffb640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 0x7f2a5406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:40.599 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.599+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a68089710 con 0x7f2a7015f8c0 2023-12-13T23:57:40.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.599+0000 7f2a754e8640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 0x7f2a5406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:40.600 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.599+0000 7f2a754e8640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 0x7f2a5406d680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c036000 tx=0x7f2a6c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:40.602 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.603+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2a680544d0 con 0x7f2a7015f8c0 2023-12-13T23:57:40.831 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.831+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f2a70144c70 con 0x7f2a7015f8c0 2023-12-13T23:57:40.832 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.833+0000 7f2a65ffb640 1 -- 172.21.15.124:0/4138206224 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) v1 ==== 184+0+141 (secure 0 0 0) 0x7f2a68003e60 con 0x7f2a7015f8c0 2023-12-13T23:57:40.833 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-12-13T23:57:40.833 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQAmRHplb8ycJBAA5eSeu2eDqEBjSnU+dDuaNw== 2023-12-13T23:57:40.833 INFO:teuthology.orchestra.run.smithi124.stdout: caps mds = "allow" 2023-12-13T23:57:40.833 INFO:teuthology.orchestra.run.smithi124.stdout: caps mon = "allow r" 2023-12-13T23:57:40.833 INFO:teuthology.orchestra.run.smithi124.stdout: caps osd = "allow rw pool=cephfs_data" 2023-12-13T23:57:40.834 INFO:teuthology.orchestra.run.smithi124.stderr:updated caps for client.0 2023-12-13T23:57:40.835 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.835+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 msgr2=0x7f2a5406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.835 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.835+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 0x7f2a5406d680 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2a6c036000 tx=0x7f2a6c03d040 comp rx=0 tx=0).stop 2023-12-13T23:57:40.835 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.835+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 msgr2=0x7f2a700c3ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:40.835 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.835+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c6e0 tx=0x7f2a6800cba0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 shutdown_connections 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2a5406b1c0 0x7f2a5406d680 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2a7015f8c0 0x7f2a700c3ed0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2a70144600 0x7f2a700cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 --2- 172.21.15.124:0/4138206224 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2a70141930 0x7f2a701f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 >> 172.21.15.124:0/4138206224 conn(0x7f2a7006d9c0 msgr2=0x7f2a700afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 shutdown_connections 2023-12-13T23:57:40.836 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:40.836+0000 7f2a7774c640 1 -- 172.21.15.124:0/4138206224 wait complete. 2023-12-13T23:57:40.845 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-12-13T23:57:40.846 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:40.885 DEBUG:teuthology.orchestra.run.smithi203:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-12-13T23:57:40.937 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-12-13T23:57:40.937 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:40.937 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:40.952 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.093+0000 7f6971c1e640 1 Processor -- start 2023-12-13T23:57:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.093+0000 7f6971c1e640 1 -- start start 2023-12-13T23:57:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6971c1e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6971c1e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 0x7f696c15d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6971c1e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0b49f0 con 0x7f696c150940 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6971c1e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0b3de0 con 0x7f696c150560 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6971c1e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0b40c0 con 0x7f696c0b3390 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f696b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6963fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 0x7f696c15d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f696b7fe640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:50366/0 (socket says 172.21.15.124:50366) 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f696bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f696c150940 0x7f696c0ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:42072/0 (socket says 172.21.15.124:42072) 2023-12-13T23:57:41.094 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f696b7fe640 1 -- 172.21.15.124:0/2431002093 learned_addr learned my addr 172.21.15.124:0/2431002093 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6963fff640 1 -- 172.21.15.124:0/2431002093 >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 msgr2=0x7f696c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6963fff640 1 --2- 172.21.15.124:0/2431002093 >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6963fff640 1 -- 172.21.15.124:0/2431002093 >> v1:172.21.15.203:6789/0 conn(0x7f696c150940 legacy=0x7f696c0ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f6963fff640 1 -- 172.21.15.124:0/2431002093 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0b43a0 con 0x7f696c0b3390 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.094+0000 7f696b7fe640 1 --2- 172.21.15.124:0/2431002093 >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.095+0000 7f6963fff640 1 --2- 172.21.15.124:0/2431002093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 0x7f696c15d200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f69540097a0 tx=0x7f695402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2f2acf7e8abac6f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.095+0000 7f696affd640 1 -- 172.21.15.124:0/2431002093 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954031830 con 0x7f696c0b3390 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.095+0000 7f696affd640 1 -- 172.21.15.124:0/2431002093 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6954031990 con 0x7f696c0b3390 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.095+0000 7f696affd640 1 -- 172.21.15.124:0/2431002093 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954031d20 con 0x7f696c0b3390 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 -- 172.21.15.124:0/2431002093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 msgr2=0x7f696c15d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 --2- 172.21.15.124:0/2431002093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 0x7f696c15d200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f69540097a0 tx=0x7f695402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 -- 172.21.15.124:0/2431002093 shutdown_connections 2023-12-13T23:57:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 --2- 172.21.15.124:0/2431002093 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0b3390 0x7f696c15d200 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 --2- 172.21.15.124:0/2431002093 >> v2:172.21.15.203:3300/0 conn(0x7f696c150560 0x7f696c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 -- 172.21.15.124:0/2431002093 >> 172.21.15.124:0/2431002093 conn(0x7f696c06a4d0 msgr2=0x7f696c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 -- 172.21.15.124:0/2431002093 shutdown_connections 2023-12-13T23:57:41.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.096+0000 7f6971c1e640 1 -- 172.21.15.124:0/2431002093 wait complete. 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.097+0000 7f6971c1e640 1 Processor -- start 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.097+0000 7f6971c1e640 1 -- start start 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 0x7f696c13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 0x7f696c157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c159250 con 0x7f696c150560 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c1efc10 con 0x7f696c0b3390 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6971c1e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0ce0b0 con 0x7f696c0cdc20 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41472/0 (socket says 172.21.15.124:41472) 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 -- 172.21.15.124:0/3684317096 learned_addr learned my addr 172.21.15.124:0/3684317096 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f696bfff640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 0x7f696c157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f696b7fe640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 0x7f696c13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 -- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 msgr2=0x7f696c157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 0x7f696c157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 -- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 msgr2=0x7f696c13f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 0x7f696c13f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 -- 172.21.15.124:0/3684317096 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c0094d0 con 0x7f696c150560 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f696bfff640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 0x7f696c157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f696b7fe640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 0x7f696c13f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.098+0000 7f6963fff640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6954009890 tx=0x7f69540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.099+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695403b980 con 0x7f696c150560 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.099+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695403bb00 con 0x7f696c150560 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.099+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6954009400 con 0x7f696c150560 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.099+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6954045400 con 0x7f696c150560 2023-12-13T23:57:41.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.099+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c158a50 con 0x7f696c150560 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.100+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c069400 con 0x7f696c150560 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.100+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f695404e430 con 0x7f696c150560 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.100+0000 7f6968ff9640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 0x7f694c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.100+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69540abc00 con 0x7f696c150560 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.101+0000 7f696b7fe640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 0x7f694c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.101+0000 7f696b7fe640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 0x7f694c06d680 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f696c14e0e0 tx=0x7f695c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.105+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f69540b2080 con 0x7f696c150560 2023-12-13T23:57:41.275 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.274+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f696c158490 con 0x7f696c150560 2023-12-13T23:57:41.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.276+0000 7f6968ff9640 1 -- 172.21.15.124:0/3684317096 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f69540769c0 con 0x7f696c150560 2023-12-13T23:57:41.276 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:41.276 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.277+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 msgr2=0x7f694c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.277+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 0x7f694c06d680 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f696c14e0e0 tx=0x7f695c008040 comp rx=0 tx=0).stop 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.277+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 msgr2=0x7f696c0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.277+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f6954009890 tx=0x7f69540046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 shutdown_connections 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f694c06b1c0 0x7f694c06d680 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f696c0cdc20 0x7f696c157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f696c150560 0x7f696c0cd6e0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 --2- 172.21.15.124:0/3684317096 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f696c0b3390 0x7f696c13f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 >> 172.21.15.124:0/3684317096 conn(0x7f696c06a4d0 msgr2=0x7f696c0ce540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 shutdown_connections 2023-12-13T23:57:41.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.278+0000 7f6971c1e640 1 -- 172.21.15.124:0/3684317096 wait complete. 2023-12-13T23:57:41.288 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:41.288 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:41.391 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.391+0000 7f19179dc640 1 Processor -- start 2023-12-13T23:57:41.391 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.391+0000 7f19179dc640 1 -- start start 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f19179dc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f1910150940 0x7f191015d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f19179dc640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100b49f0 con 0x7f191015dec0 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f19179dc640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19100b3560 con 0x7f1910150560 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f19179dc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19100b3de0 con 0x7f1910150940 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f1915f79640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f191015dec0 0x7f19100ce0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34558/0 (socket says 172.21.15.124:34558) 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.392+0000 7f1915f79640 1 -- 172.21.15.124:0/2150751161 learned_addr learned my addr 172.21.15.124:0/2150751161 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2643054638 0 0) 0x7f19100b49f0 con 0x7f191015dec0 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18fc003520 con 0x7f191015dec0 2023-12-13T23:57:41.395 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1914f77640 1 --2- 172.21.15.124:0/2150751161 >> v2:172.21.15.203:3300/0 conn(0x7f1910150940 0x7f191015d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2844692784 0 0) 0x7f18fc003520 con 0x7f191015dec0 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 >> v2:172.21.15.203:3300/0 conn(0x7f1910150940 msgr2=0x7f191015d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 --2- 172.21.15.124:0/2150751161 >> v2:172.21.15.203:3300/0 conn(0x7f1910150940 0x7f191015d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 >> v1:172.21.15.203:6789/0 conn(0x7f1910150560 legacy=0x7f19100b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19100b40c0 con 0x7f191015dec0 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f1908001670 con 0x7f191015dec0 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 >> v1:172.21.15.124:6789/0 conn(0x7f191015dec0 legacy=0x7f19100ce0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18fc003970 0x7f18fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18fc0047c0 0x7f18fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f18fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19100b3de0 con 0x7f18fc005180 2023-12-13T23:57:41.396 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc00c020 con 0x7f18fc0047c0 2023-12-13T23:57:41.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc00c1e0 con 0x7f18fc003970 2023-12-13T23:57:41.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f18fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1914f77640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18fc0047c0 0x7f18fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915778640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18fc003970 0x7f18fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 -- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18fc003970 msgr2=0x7f18fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18fc003970 0x7f18fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 -- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18fc0047c0 msgr2=0x7f18fc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18fc0047c0 0x7f18fc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915f79640 1 -- 172.21.15.124:0/2150751161 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc003520 con 0x7f18fc005180 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1915778640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f18fc003970 0x7f18fc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.393+0000 7f1914f77640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f18fc0047c0 0x7f18fc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.394+0000 7f1915f79640 1 -- 172.21.15.124:0/2150751161 reap_dead start 2023-12-13T23:57:41.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.394+0000 7f1915f79640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f18fc00b980 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1908001050 tx=0x7f18fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19c122b546c9883 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.394+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1908005ad0 con 0x7f18fc005180 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.394+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1908005c30 con 0x7f18fc005180 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.394+0000 7f1907fff640 1 -- 172.21.15.124:0/2150751161 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190802c890 con 0x7f18fc005180 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 -- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 msgr2=0x7f18fc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f18fc00b980 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1908001050 tx=0x7f18fc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 -- 172.21.15.124:0/2150751161 shutdown_connections 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 --2- 172.21.15.124:0/2150751161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f18fc00b980 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 -- 172.21.15.124:0/2150751161 >> 172.21.15.124:0/2150751161 conn(0x7f191006a4d0 msgr2=0x7f191006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 -- 172.21.15.124:0/2150751161 shutdown_connections 2023-12-13T23:57:41.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.395+0000 7f19179dc640 1 -- 172.21.15.124:0/2150751161 wait complete. 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.396+0000 7f19179dc640 1 Processor -- start 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.396+0000 7f19179dc640 1 -- start start 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1910070290 0x7f1910070710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1910141f10 0x7f19101423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19101406d0 con 0x7f18fc005180 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19100ad870 con 0x7f1910070290 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1910070c50 con 0x7f1910141f10 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.402 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41484/0 (socket says 172.21.15.124:41484) 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 -- 172.21.15.124:0/3033736320 learned_addr learned my addr 172.21.15.124:0/3033736320 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1914f77640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1910070290 0x7f1910070710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915f79640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1910141f10 0x7f19101423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 -- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1910141f10 msgr2=0x7f19101423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1910141f10 0x7f19101423c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 -- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1910070290 msgr2=0x7f1910070710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1910070290 0x7f1910070710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 -- 172.21.15.124:0/3033736320 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1910140030 con 0x7f18fc005180 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1915778640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f190c00e7f0 tx=0x7f190c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c003b80 con 0x7f18fc005180 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f190c003ce0 con 0x7f18fc005180 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1910142c50 con 0x7f18fc005180 2023-12-13T23:57:41.403 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.397+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190c005310 con 0x7f18fc005180 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.398+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1910142900 con 0x7f18fc005180 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.398+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1910069400 con 0x7f18fc005180 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.399+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f190c005470 con 0x7f18fc005180 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.399+0000 7f1905ffb640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 0x7f18f406d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.399+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f190c089a30 con 0x7f18fc005180 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.399+0000 7f1914f77640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 0x7f18f406d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.400+0000 7f1914f77640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 0x7f18f406d6b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f19100acc10 tx=0x7f1900008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.403+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f190c054770 con 0x7f18fc005180 2023-12-13T23:57:41.571 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.572+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f191006aa50 con 0x7f18fc005180 2023-12-13T23:57:41.572 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.572+0000 7f1905ffb640 1 -- 172.21.15.124:0/3033736320 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f190c057d90 con 0x7f18fc005180 2023-12-13T23:57:41.572 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 msgr2=0x7f18f406d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 0x7f18f406d6b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f19100acc10 tx=0x7f1900008040 comp rx=0 tx=0).stop 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 msgr2=0x7f191013f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f190c00e7f0 tx=0x7f190c00ecb0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 shutdown_connections 2023-12-13T23:57:41.574 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f18f406b1f0 0x7f18f406d6b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f1910070290 0x7f1910070710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f1910141f10 0x7f19101423c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 --2- 172.21.15.124:0/3033736320 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18fc005180 0x7f191013f570 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 >> 172.21.15.124:0/3033736320 conn(0x7f191006a4d0 msgr2=0x7f1910071290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 shutdown_connections 2023-12-13T23:57:41.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.574+0000 7f19179dc640 1 -- 172.21.15.124:0/3033736320 wait complete. 2023-12-13T23:57:41.584 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:41.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 Processor -- start 2023-12-13T23:57:41.686 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 -- start start 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f503806ea50 0x7f503806ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380d1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5038161d20 con 0x7f5038160fb0 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503806f370 con 0x7f503806ea50 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.687+0000 7f503e0d0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503815e760 con 0x7f50380d0ed0 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50377fe640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f503806ea50 0x7f503806ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f5036ffd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f5038160fb0 0x7f5038161440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34562/0 (socket says 172.21.15.124:34562) 2023-12-13T23:57:41.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f5036ffd640 1 -- 172.21.15.124:0/1485482415 learned_addr learned my addr 172.21.15.124:0/1485482415 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f5037fff640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380d1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50377fe640 1 -- 172.21.15.124:0/1485482415 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 msgr2=0x7f50380d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50377fe640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380d1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50377fe640 1 -- 172.21.15.124:0/1485482415 >> v1:172.21.15.124:6789/0 conn(0x7f5038160fb0 legacy=0x7f5038161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50377fe640 1 -- 172.21.15.124:0/1485482415 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503815d9a0 con 0x7f503806ea50 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.688+0000 7f50367fc640 1 -- 172.21.15.124:0/1485482415 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109289702 0 0) 0x7f5038161d20 con 0x7f5038160fb0 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f50377fe640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f503806ee30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f50280097a0 tx=0x7f502802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5652f6a3e832968d server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f50367fc640 1 -- 172.21.15.124:0/1485482415 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028031830 con 0x7f503806ea50 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f50367fc640 1 -- 172.21.15.124:0/1485482415 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5028031990 con 0x7f503806ea50 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f50367fc640 1 -- 172.21.15.124:0/1485482415 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028031d20 con 0x7f503806ea50 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 -- 172.21.15.124:0/1485482415 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 msgr2=0x7f503806ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f503806ee30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f50280097a0 tx=0x7f502802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 -- 172.21.15.124:0/1485482415 shutdown_connections 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380d1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 --2- 172.21.15.124:0/1485482415 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f503806ee30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 -- 172.21.15.124:0/1485482415 >> 172.21.15.124:0/1485482415 conn(0x7f503806a6b0 msgr2=0x7f503806aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 -- 172.21.15.124:0/1485482415 shutdown_connections 2023-12-13T23:57:41.691 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.689+0000 7f503e0d0640 1 -- 172.21.15.124:0/1485482415 wait complete. 2023-12-13T23:57:41.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.690+0000 7f503e0d0640 1 Processor -- start 2023-12-13T23:57:41.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 -- start start 2023-12-13T23:57:41.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 0x7f50380b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50381f62f0 con 0x7f503806ea50 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50380cb320 con 0x7f5038160fb0 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f503e0d0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50380cb600 con 0x7f50380d0ed0 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f50377fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f50377fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41492/0 (socket says 172.21.15.124:41492) 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.691+0000 7f50377fe640 1 -- 172.21.15.124:0/97765790 learned_addr learned my addr 172.21.15.124:0/97765790 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f5036ffd640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f5037fff640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 0x7f50380b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 -- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 msgr2=0x7f50380b42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380b42c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 -- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 msgr2=0x7f50380b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 0x7f50380b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 -- 172.21.15.124:0/97765790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028009400 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f5037fff640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 0x7f50380b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f5036ffd640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f50377fe640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f50280097a0 tx=0x7f50280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5028004230 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5028004390 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50380cb8e0 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502803a930 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.692+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50381f26e0 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.693+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50381f2930 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.693+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f502803aa90 con 0x7f503806ea50 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.694+0000 7f501ffff640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 0x7f502406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.694+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50280abc70 con 0x7f503806ea50 2023-12-13T23:57:41.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.694+0000 7f5036ffd640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 0x7f502406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.695+0000 7f5036ffd640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 0x7f502406d680 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f50381f5f20 tx=0x7f502c009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.698+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f5028076b50 con 0x7f503806ea50 2023-12-13T23:57:41.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.867+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5038159850 con 0x7f503806ea50 2023-12-13T23:57:41.867 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.867+0000 7f501ffff640 1 -- 172.21.15.124:0/97765790 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50280350a0 con 0x7f503806ea50 2023-12-13T23:57:41.867 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 msgr2=0x7f502406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 0x7f502406d680 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f50381f5f20 tx=0x7f502c009040 comp rx=0 tx=0).stop 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 msgr2=0x7f50381f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f50280097a0 tx=0x7f50280046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 shutdown_connections 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f502406b1c0 0x7f502406d680 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f5038160fb0 0x7f50380b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f50380d0ed0 0x7f50380b42c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.869 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 --2- 172.21.15.124:0/97765790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f503806ea50 0x7f50381f21a0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 >> 172.21.15.124:0/97765790 conn(0x7f503806a6b0 msgr2=0x7f503815b420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 shutdown_connections 2023-12-13T23:57:41.870 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.869+0000 7f503e0d0640 1 -- 172.21.15.124:0/97765790 wait complete. 2023-12-13T23:57:41.880 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:41.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.983+0000 7fb33122b640 1 Processor -- start 2023-12-13T23:57:41.982 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.983+0000 7fb33122b640 1 -- start start 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb33122b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb32c0b1400 0x7fb32c15a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb33122b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 0x7fb32c15aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb33122b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32c15e2a0 con 0x7fb32c143490 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb33122b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb32c15b520 con 0x7fb32c0b1400 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb33122b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c143870 con 0x7fb32c15e730 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb32c0b1400 0x7fb32c15a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb32c0b1400 0x7fb32c15a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41508/0 (socket says 172.21.15.124:41508) 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 -- 172.21.15.124:0/3184055342 learned_addr learned my addr 172.21.15.124:0/3184055342 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32ad9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fb32c143490 0x7fb32c0b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34572/0 (socket says 172.21.15.124:34572) 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32b59e640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 0x7fb32c15aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 -- 172.21.15.124:0/3184055342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 msgr2=0x7fb32c15aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 0x7fb32c15aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.984 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 -- 172.21.15.124:0/3184055342 >> v1:172.21.15.124:6789/0 conn(0x7fb32c143490 legacy=0x7fb32c0b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32a59c640 1 -- 172.21.15.124:0/3184055342 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c15b240 con 0x7fb32c0b1400 2023-12-13T23:57:41.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb329d9b640 1 -- 172.21.15.124:0/3184055342 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274950704 0 0) 0x7fb32c15e2a0 con 0x7fb32c143490 2023-12-13T23:57:41.985 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.984+0000 7fb32b59e640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 0x7fb32c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.985+0000 7fb32a59c640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c0b1400 0x7fb32c15a4e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb32000bb10 tx=0x7fb32002e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8d63bc70e8f9f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.985+0000 7fb329d9b640 1 -- 172.21.15.124:0/3184055342 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320036440 con 0x7fb32c0b1400 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.985+0000 7fb329d9b640 1 -- 172.21.15.124:0/3184055342 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3200365a0 con 0x7fb32c0b1400 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.985+0000 7fb329d9b640 1 -- 172.21.15.124:0/3184055342 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb320036930 con 0x7fb32c0b1400 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 -- 172.21.15.124:0/3184055342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c0b1400 msgr2=0x7fb32c15a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c0b1400 0x7fb32c15a4e0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb32000bb10 tx=0x7fb32002e150 comp rx=0 tx=0).stop 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 -- 172.21.15.124:0/3184055342 shutdown_connections 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c15e730 0x7fb32c15aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 --2- 172.21.15.124:0/3184055342 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c0b1400 0x7fb32c15a4e0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 -- 172.21.15.124:0/3184055342 >> 172.21.15.124:0/3184055342 conn(0x7fb32c06a4d0 msgr2=0x7fb32c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:41.986 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 -- 172.21.15.124:0/3184055342 shutdown_connections 2023-12-13T23:57:41.987 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.986+0000 7fb33122b640 1 -- 172.21.15.124:0/3184055342 wait complete. 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.987+0000 7fb33122b640 1 Processor -- start 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.987+0000 7fb33122b640 1 -- start start 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 0x7fb32c0ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 0x7fb32c13c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.989 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0a8960 con 0x7fb32c15e730 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0a8c40 con 0x7fb32c143490 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb33122b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32c0a8f20 con 0x7fb32c0b1400 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41524/0 (socket says 172.21.15.124:41524) 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 -- 172.21.15.124:0/416033539 learned_addr learned my addr 172.21.15.124:0/416033539 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32ad9d640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 0x7fb32c0ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32a59c640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 0x7fb32c13c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 -- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 msgr2=0x7fb32c0ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 0x7fb32c0ae640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 -- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 msgr2=0x7fb32c13c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 0x7fb32c13c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 -- 172.21.15.124:0/416033539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32000b770 con 0x7fb32c15e730 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32a59c640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 0x7fb32c13c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32ad9d640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 0x7fb32c0ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:41.990 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.988+0000 7fb32b59e640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb32400cd50 tx=0x7fb32400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.989+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32400f680 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.989+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32400f800 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.989+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb32c0a9200 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.989+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb32400fbb0 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.989+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb32c0ab7e0 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.990+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb32c069400 con 0x7fb32c15e730 2023-12-13T23:57:41.991 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.990+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fb32400b040 con 0x7fb32c15e730 2023-12-13T23:57:41.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.990+0000 7fb3137fe640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 0x7fb31806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:41.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.991+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb32408a430 con 0x7fb32c15e730 2023-12-13T23:57:41.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.991+0000 7fb32ad9d640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 0x7fb31806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:41.992 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.991+0000 7fb32ad9d640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 0x7fb31806d680 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb31c000f60 tx=0x7fb31c007c10 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:41.994 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:41.995+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fb324055160 con 0x7fb32c15e730 2023-12-13T23:57:42.158 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.159+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb32c0a8aa0 con 0x7fb32c15e730 2023-12-13T23:57:42.159 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.159+0000 7fb3137fe640 1 -- 172.21.15.124:0/416033539 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb32c0a8aa0 con 0x7fb32c15e730 2023-12-13T23:57:42.159 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 msgr2=0x7fb31806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 0x7fb31806d680 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb31c000f60 tx=0x7fb31c007c10 comp rx=0 tx=0).stop 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 msgr2=0x7fb32c0ce300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fb32400cd50 tx=0x7fb32400a650 comp rx=0 tx=0).stop 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 shutdown_connections 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fb31806b1c0 0x7fb31806d680 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fb32c143490 0x7fb32c13c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb32c15e730 0x7fb32c0ce300 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.161 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 --2- 172.21.15.124:0/416033539 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fb32c0b1400 0x7fb32c0ae640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 >> 172.21.15.124:0/416033539 conn(0x7fb32c06a4d0 msgr2=0x7fb32c142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:42.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 shutdown_connections 2023-12-13T23:57:42.162 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.161+0000 7fb33122b640 1 -- 172.21.15.124:0/416033539 wait complete. 2023-12-13T23:57:42.171 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-12-13T23:57:42.272 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.273+0000 7fd025389640 1 Processor -- start 2023-12-13T23:57:42.273 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.273+0000 7fd025389640 1 -- start start 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd025389640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd0200cdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd025389640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fd0200ce270 0x7fd0200d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd025389640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0200b3450 con 0x7fd020160bb0 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd025389640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0200d4140 con 0x7fd0200ce270 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd025389640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b45b0 con 0x7fd0200b3fc0 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01effd640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd020160bb0 0x7fd0200d3a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34580/0 (socket says 172.21.15.124:34580) 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01effd640 1 -- 172.21.15.124:0/4242284465 learned_addr learned my addr 172.21.15.124:0/4242284465 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd017fff640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd0200cdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01f7fe640 1 --2- 172.21.15.124:0/4242284465 >> v2:172.21.15.203:3300/0 conn(0x7fd0200ce270 0x7fd0200d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637750146 0 0) 0x7fd0200b3450 con 0x7fd020160bb0 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd00c003520 con 0x7fd020160bb0 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3936908042 0 0) 0x7fd00c003520 con 0x7fd020160bb0 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 msgr2=0x7fd0200cdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd0200cdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 >> v2:172.21.15.203:3300/0 conn(0x7fd0200ce270 msgr2=0x7fd0200d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 --2- 172.21.15.124:0/4242284465 >> v2:172.21.15.203:3300/0 conn(0x7fd0200ce270 0x7fd0200d1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0200b4890 con 0x7fd020160bb0 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fd010001670 con 0x7fd020160bb0 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 >> v1:172.21.15.124:6789/0 conn(0x7fd020160bb0 legacy=0x7fd0200d3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.274+0000 7fd01e7fc640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd00c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01e7fc640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd00c004d00 0x7fd00c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200d4140 con 0x7fd00c004d00 2023-12-13T23:57:42.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200b45b0 con 0x7fd00c003970 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c00b980 con 0x7fd0200b3fc0 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd00c004d00 0x7fd00c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01effd640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd017fff640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd00c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 -- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 msgr2=0x7fd00c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200b3fc0 0x7fd00c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 -- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd00c003970 msgr2=0x7fd00c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd00c003970 0x7fd00c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.275+0000 7fd01f7fe640 1 -- 172.21.15.124:0/4242284465 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c003520 con 0x7fd00c004d00 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd01f7fe640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd00c004d00 0x7fd00c005180 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd01800e170 tx=0x7fd00c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b624e11d882e12 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd01802cc20 con 0x7fd00c004d00 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd01800f040 con 0x7fd00c004d00 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd01e7fc640 1 -- 172.21.15.124:0/4242284465 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd018034440 con 0x7fd00c004d00 2023-12-13T23:57:42.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 -- 172.21.15.124:0/4242284465 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd00c004d00 msgr2=0x7fd00c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 --2- 172.21.15.124:0/4242284465 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd00c004d00 0x7fd00c005180 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fd01800e170 tx=0x7fd00c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd01f7fe640 1 -- 172.21.15.124:0/4242284465 reap_dead start 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 -- 172.21.15.124:0/4242284465 shutdown_connections 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 -- 172.21.15.124:0/4242284465 >> 172.21.15.124:0/4242284465 conn(0x7fd02009c590 msgr2=0x7fd0200d3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 -- 172.21.15.124:0/4242284465 shutdown_connections 2023-12-13T23:57:42.279 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.276+0000 7fd025389640 1 -- 172.21.15.124:0/4242284465 wait complete. 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 Processor -- start 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 -- start start 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200cba00 0x7fd02009db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0201f2f20 0x7fd0201f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02015afe0 con 0x7fd02009eb30 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02009f4f0 con 0x7fd0201f2f20 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd025389640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0200a0fc0 con 0x7fd0200cba00 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd017fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd017fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41534/0 (socket says 172.21.15.124:41534) 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.278+0000 7fd017fff640 1 -- 172.21.15.124:0/4191985350 learned_addr learned my addr 172.21.15.124:0/4191985350 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:42.282 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 -- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200cba00 msgr2=0x7fd02009db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200cba00 0x7fd02009db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 -- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0201f2f20 msgr2=0x7fd0201f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0201f2f20 0x7fd0201f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 -- 172.21.15.124:0/4191985350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0201f3e50 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd017fff640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd008011940 tx=0x7fd008011e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd00800fa70 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd00800fbd0 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd008014440 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0201f38e0 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.279+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02009d030 con 0x7fd02009eb30 2023-12-13T23:57:42.283 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.280+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd020069400 con 0x7fd02009eb30 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.280+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fd0080145a0 con 0x7fd02009eb30 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.281+0000 7fd0177fe640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 0x7fd00006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.281+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd00808a1a0 con 0x7fd02009eb30 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.281+0000 7fd01effd640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 0x7fd00006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.281+0000 7fd01effd640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 0x7fd00006d6b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd010001cc0 tx=0x7fd010001c30 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.285+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fd008054f60 con 0x7fd02009eb30 2023-12-13T23:57:42.349 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:57:42.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:57:42.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.469+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd0200d3430 con 0x7fd02009eb30 2023-12-13T23:57:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.469+0000 7fd0177fe640 1 -- 172.21.15.124:0/4191985350 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v50) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd008058580 con 0x7fd02009eb30 2023-12-13T23:57:42.469 INFO:teuthology.orchestra.run.smithi124.stderr:listed 0 entries 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 msgr2=0x7fd00006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 0x7fd00006d6b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd010001cc0 tx=0x7fd010001c30 comp rx=0 tx=0).stop 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 msgr2=0x7fd02009efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fd008011940 tx=0x7fd008011e00 comp rx=0 tx=0).stop 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 shutdown_connections 2023-12-13T23:57:42.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fd00006b1f0 0x7fd00006d6b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fd0201f2f20 0x7fd0201f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd02009eb30 0x7fd02009efb0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 --2- 172.21.15.124:0/4191985350 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fd0200cba00 0x7fd02009db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 >> 172.21.15.124:0/4191985350 conn(0x7fd02009c590 msgr2=0x7fd0200b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.471+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 shutdown_connections 2023-12-13T23:57:42.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.472+0000 7fd025389640 1 -- 172.21.15.124:0/4191985350 wait complete. 2023-12-13T23:57:42.481 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2023-12-13T23:57:42.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:42.482 DEBUG:teuthology.orchestra.run.smithi124:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-12-13T23:57:42.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[110821]: fuse finished with error 0 and tester_r 0 2023-12-13T23:57:42.508 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:42.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:42.508 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-12-13T23:57:42.599 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:42.600 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:42.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:42.618 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:57:42.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:42.619 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-12-13T23:57:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:42.677 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2023-12-13T23:57:42.677 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:42.677 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:42.693 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:42.693 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:57:42.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:42.693 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-12-13T23:57:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:42.748 INFO:teuthology.orchestra.run.smithi124.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2023-12-13T23:57:42.748 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:42.748 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:42.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:42.767 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:57:42.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:42.767 DEBUG:teuthology.orchestra.run.smithi203:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-12-13T23:57:42.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:57:42.826 INFO:teuthology.orchestra.run.smithi203.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2023-12-13T23:57:42.826 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:42.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.927+0000 7f717ec90640 1 Processor -- start 2023-12-13T23:57:42.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.927+0000 7f717ec90640 1 -- start start 2023-12-13T23:57:42.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ec90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f71780b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ec90640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f717815abf0 0x7f71780b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ec90640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717815e2a0 con 0x7f717815e730 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ec90640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f717815b520 con 0x7f717815abf0 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ec90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7178143870 con 0x7f7178143490 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f7177fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f717815e730 0x7f717815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34582/0 (socket says 172.21.15.124:34582) 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f7177fff640 1 -- 172.21.15.124:0/2888990433 learned_addr learned my addr 172.21.15.124:0/2888990433 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745373439 0 0) 0x7f717815e2a0 con 0x7f717815e730 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7168003520 con 0x7f717815e730 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717ca2c640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f71780b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717d22d640 1 --2- 172.21.15.124:0/2888990433 >> v2:172.21.15.203:3300/0 conn(0x7f717815abf0 0x7f71780b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3296561614 0 0) 0x7f7168003520 con 0x7f717815e730 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 msgr2=0x7f71780b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f71780b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 >> v2:172.21.15.203:3300/0 conn(0x7f717815abf0 msgr2=0x7f71780b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 --2- 172.21.15.124:0/2888990433 >> v2:172.21.15.203:3300/0 conn(0x7f717815abf0 0x7f71780b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717815b240 con 0x7f717815e730 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f7164001670 con 0x7f717815e730 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 >> v1:172.21.15.124:6789/0 conn(0x7f717815e730 legacy=0x7f717815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f7168003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7168003fa0 0x7f71680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7168004d00 0x7f7168005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717815b520 con 0x7f7168004d00 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7178143870 con 0x7f7168003fa0 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716800ba80 con 0x7f7178143490 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.928+0000 7f717d22d640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7168004d00 0x7f7168005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f7177fff640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7168003fa0 0x7f71680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717ca2c640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f7168003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 -- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 msgr2=0x7f7168003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f7168003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 -- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7168003fa0 msgr2=0x7f71680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7168003fa0 0x7f71680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 -- 172.21.15.124:0/2888990433 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7168003520 con 0x7f7168004d00 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717ca2c640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7178143490 0x7f7168003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.929+0000 7f717d22d640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7168004d00 0x7f7168005180 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7170007360 tx=0x7f71680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8238d72aba8b42e7 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717002ccf0 con 0x7f7168004d00 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717002ce50 con 0x7f7168004d00 2023-12-13T23:57:42.938 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f71777fe640 1 -- 172.21.15.124:0/2888990433 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170034660 con 0x7f7168004d00 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 -- 172.21.15.124:0/2888990433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7168004d00 msgr2=0x7f7168005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 --2- 172.21.15.124:0/2888990433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7168004d00 0x7f7168005180 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f7170007360 tx=0x7f71680057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717d22d640 1 -- 172.21.15.124:0/2888990433 reap_dead start 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 -- 172.21.15.124:0/2888990433 shutdown_connections 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 -- 172.21.15.124:0/2888990433 >> 172.21.15.124:0/2888990433 conn(0x7f717806a4d0 msgr2=0x7f717806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 -- 172.21.15.124:0/2888990433 shutdown_connections 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.930+0000 7f717ec90640 1 -- 172.21.15.124:0/2888990433 wait complete. 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.931+0000 7f717ec90640 1 Processor -- start 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.931+0000 7f717ec90640 1 -- start start 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7168004d00 0x7f71781596a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 0x7f71781386b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717813e470 con 0x7f717813aa70 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7178139550 con 0x7f717813a5f0 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717ec90640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7178139830 con 0x7f7168004d00 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717d22d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717d22d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41556/0 (socket says 172.21.15.124:41556) 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.932+0000 7f717d22d640 1 -- 172.21.15.124:0/3375681479 learned_addr learned my addr 172.21.15.124:0/3375681479 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717ca2c640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7168004d00 0x7f71781596a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f7177fff640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 0x7f71781386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 -- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7168004d00 msgr2=0x7f71781596a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7168004d00 0x7f71781596a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 -- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 msgr2=0x7f71781386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 0x7f71781386b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:42.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 -- 172.21.15.124:0/3375681479 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178139b10 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f7177fff640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 0x7f71781386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717d22d640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f717002dfd0 tx=0x7f7170004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170045070 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170034e60 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7178139df0 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717003d620 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.933+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7178159be0 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.934+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7178069400 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.934+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f717003b040 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.935+0000 7f71757fa640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 0x7f715c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.935+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f71700aa9c0 con 0x7f717813aa70 2023-12-13T23:57:42.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.935+0000 7f717ca2c640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 0x7f715c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:42.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.935+0000 7f717ca2c640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 0x7f715c06d680 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f716c006fd0 tx=0x7f716c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:42.942 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:42.939+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7170075780 con 0x7f717813aa70 2023-12-13T23:57:43.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.141+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71780b2820 con 0x7f717813aa70 2023-12-13T23:57:43.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.142+0000 7f71757fa640 1 -- 172.21.15.124:0/3375681479 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f7170039090 con 0x7f717813aa70 2023-12-13T23:57:43.141 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:43.142 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:43.142 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:43.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 msgr2=0x7f715c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 0x7f715c06d680 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f716c006fd0 tx=0x7f716c008040 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 msgr2=0x7f7178138bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f717002dfd0 tx=0x7f7170004ab0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 shutdown_connections 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f715c06b1c0 0x7f715c06d680 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f717813a5f0 0x7f71781386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f717813aa70 0x7f7178138bf0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 --2- 172.21.15.124:0/3375681479 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7168004d00 0x7f71781596a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 >> 172.21.15.124:0/3375681479 conn(0x7f717806a4d0 msgr2=0x7f71780cdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 shutdown_connections 2023-12-13T23:57:43.144 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.144+0000 7f717ec90640 1 -- 172.21.15.124:0/3375681479 wait complete. 2023-12-13T23:57:43.154 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:43.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.255+0000 7f7fa6f61640 1 Processor -- start 2023-12-13T23:57:43.255 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.255+0000 7f7fa6f61640 1 -- start start 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.256+0000 7f7fa6f61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7fa00d2910 0x7f7fa00d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.256+0000 7f7fa6f61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 0x7f7fa00b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.256+0000 7f7fa6f61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa00b2b10 con 0x7f7fa015d330 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.256+0000 7f7fa6f61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa00d1590 con 0x7f7fa00d2910 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.256+0000 7f7fa6f61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa00d1870 con 0x7f7fa00d3730 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7fa00d2910 0x7f7fa00d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7fa54fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 0x7f7fa00b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7fa00d2910 0x7f7fa00d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:50462/0 (socket says 172.21.15.124:50462) 2023-12-13T23:57:43.256 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 -- 172.21.15.124:0/2637405924 learned_addr learned my addr 172.21.15.124:0/2637405924 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 -- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 msgr2=0x7f7fa00b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 0x7f7fa00b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 -- 172.21.15.124:0/2637405924 >> v1:172.21.15.203:6789/0 conn(0x7f7fa015d330 legacy=0x7f7fa00b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7f9ffff640 1 -- 172.21.15.124:0/2637405924 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa00d3b90 con 0x7f7fa00d2910 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.257+0000 7f7fa54fe640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 0x7f7fa00b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.258+0000 7f7f9ffff640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa00d31f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009870 tx=0x7f7f9402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53dfaeeab52c2d90 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.258+0000 7f7f9f7fe640 1 -- 172.21.15.124:0/2637405924 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94031640 con 0x7f7fa00d2910 2023-12-13T23:57:43.258 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.258+0000 7f7f9f7fe640 1 -- 172.21.15.124:0/2637405924 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f940317a0 con 0x7f7fa00d2910 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.258+0000 7f7f9f7fe640 1 -- 172.21.15.124:0/2637405924 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94031b30 con 0x7f7fa00d2910 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 -- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 msgr2=0x7f7fa00d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa00d31f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009870 tx=0x7f7f9402b8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 -- 172.21.15.124:0/2637405924 shutdown_connections 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa00d3730 0x7f7fa00b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/2637405924 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa00d31f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 -- 172.21.15.124:0/2637405924 >> 172.21.15.124:0/2637405924 conn(0x7f7fa0072020 msgr2=0x7f7fa0072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 -- 172.21.15.124:0/2637405924 shutdown_connections 2023-12-13T23:57:43.259 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.259+0000 7f7fa6f61640 1 -- 172.21.15.124:0/2637405924 wait complete. 2023-12-13T23:57:43.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.260+0000 7f7fa6f61640 1 Processor -- start 2023-12-13T23:57:43.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.260+0000 7f7fa6f61640 1 -- start start 2023-12-13T23:57:43.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa014fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.261 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 0x7f7fa014cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa015f200 con 0x7f7fa00d3730 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa015f4a0 con 0x7f7fa00d2910 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa6f61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa015dd60 con 0x7f7fa015d330 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41572/0 (socket says 172.21.15.124:41572) 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 -- 172.21.15.124:0/1398979891 learned_addr learned my addr 172.21.15.124:0/1398979891 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa4cfd640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa014fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa54fe640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 0x7f7fa014cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 -- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 msgr2=0x7f7fa014cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 0x7f7fa014cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 -- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 msgr2=0x7f7fa014fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa014fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7f9ffff640 1 -- 172.21.15.124:0/1398979891 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f98009a70 con 0x7f7fa00d3730 2023-12-13T23:57:43.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.261+0000 7f7fa54fe640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 0x7f7fa014cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7f9ffff640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7f94000c00 tx=0x7f7f940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9403a930 con 0x7f7fa00d3730 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9403aa90 con 0x7f7fa00d3730 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f940094d0 con 0x7f7fa00d3730 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9403ae40 con 0x7f7fa00d3730 2023-12-13T23:57:43.263 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.262+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa015f740 con 0x7f7fa00d3730 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.263+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa0159f60 con 0x7f7fa00d3730 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.263+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7f9402f050 con 0x7f7fa00d3730 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.264+0000 7f7f9d7fa640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 0x7f7f8c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.264+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7f940ab880 con 0x7f7fa00d3730 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.264+0000 7f7fa4cfd640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 0x7f7f8c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.264+0000 7f7fa4cfd640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 0x7f7f8c06d680 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000f60 tx=0x7f7f90009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.267 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.268+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7f940765b0 con 0x7f7fa00d3730 2023-12-13T23:57:43.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.434+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fa00693e0 con 0x7f7fa00d3730 2023-12-13T23:57:43.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.436+0000 7f7f9d7fa640 1 -- 172.21.15.124:0/1398979891 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f7f94079bd0 con 0x7f7fa00d3730 2023-12-13T23:57:43.435 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:43.435 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.437+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 msgr2=0x7f7f8c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.437+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 0x7f7f8c06d680 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f7f90000f60 tx=0x7f7f90009040 comp rx=0 tx=0).stop 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.437+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 msgr2=0x7f7fa014c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.437+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f7f94000c00 tx=0x7f7f940046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 shutdown_connections 2023-12-13T23:57:43.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7f8c06b1c0 0x7f7f8c06d680 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7fa015d330 0x7f7fa014cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7fa00d3730 0x7f7fa014c540 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 --2- 172.21.15.124:0/1398979891 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7fa00d2910 0x7f7fa014fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 >> 172.21.15.124:0/1398979891 conn(0x7f7fa0072020 msgr2=0x7f7fa0072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 shutdown_connections 2023-12-13T23:57:43.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.438+0000 7f7fa6f61640 1 -- 172.21.15.124:0/1398979891 wait complete. 2023-12-13T23:57:43.448 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-12-13T23:55:00.158578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2023-12-13T23:57:15.517529+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '49', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2023-12-13T23:57:16.303275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '50', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-12-13T23:57:43.448 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-12-13T23:57:43.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.551+0000 7fdae837d640 1 Processor -- start 2023-12-13T23:57:43.551 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 -- start start 2023-12-13T23:57:43.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdae0144980 0x7fdae015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fdae00b4370 0x7fdae00b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.552 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae0071770 con 0x7fdae00ceb70 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae00ccb60 con 0x7fdae0144980 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.552+0000 7fdae837d640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae00b4ce0 con 0x7fdae00b4370 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae6119640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdae0144980 0x7fdae015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae5918640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdae00ceb70 0x7fdae00cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34598/0 (socket says 172.21.15.124:34598) 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae5918640 1 -- 172.21.15.124:0/2661305112 learned_addr learned my addr 172.21.15.124:0/2661305112 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae6119640 1 -- 172.21.15.124:0/2661305112 >> v2:172.21.15.203:3300/0 conn(0x7fdae00b4370 msgr2=0x7fdae00b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae691a640 1 --2- 172.21.15.124:0/2661305112 >> v2:172.21.15.203:3300/0 conn(0x7fdae00b4370 0x7fdae00b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae6119640 1 --2- 172.21.15.124:0/2661305112 >> v2:172.21.15.203:3300/0 conn(0x7fdae00b4370 0x7fdae00b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae6119640 1 -- 172.21.15.124:0/2661305112 >> v1:172.21.15.124:6789/0 conn(0x7fdae00ceb70 legacy=0x7fdae00cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae5117640 1 -- 172.21.15.124:0/2661305112 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416951626 0 0) 0x7fdae0071770 con 0x7fdae00ceb70 2023-12-13T23:57:43.553 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.553+0000 7fdae6119640 1 -- 172.21.15.124:0/2661305112 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae00ccdc0 con 0x7fdae0144980 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae6119640 1 --2- 172.21.15.124:0/2661305112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0144980 0x7fdae015c7c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdae00716e0 tx=0x7fdadc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99172c3e1c592e25 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae5117640 1 -- 172.21.15.124:0/2661305112 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc031830 con 0x7fdae0144980 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae5117640 1 -- 172.21.15.124:0/2661305112 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc031990 con 0x7fdae0144980 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae5117640 1 -- 172.21.15.124:0/2661305112 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc031d20 con 0x7fdae0144980 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 -- 172.21.15.124:0/2661305112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0144980 msgr2=0x7fdae015c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 --2- 172.21.15.124:0/2661305112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0144980 0x7fdae015c7c0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fdae00716e0 tx=0x7fdadc02b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 -- 172.21.15.124:0/2661305112 shutdown_connections 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 --2- 172.21.15.124:0/2661305112 >> v2:172.21.15.203:3300/0 conn(0x7fdae00b4370 0x7fdae00b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 --2- 172.21.15.124:0/2661305112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae0144980 0x7fdae015c7c0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 -- 172.21.15.124:0/2661305112 >> 172.21.15.124:0/2661305112 conn(0x7fdae006a4d0 msgr2=0x7fdae006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.555 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 -- 172.21.15.124:0/2661305112 shutdown_connections 2023-12-13T23:57:43.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.554+0000 7fdae837d640 1 -- 172.21.15.124:0/2661305112 wait complete. 2023-12-13T23:57:43.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 Processor -- start 2023-12-13T23:57:43.556 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 -- start start 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdae00ceb70 0x7fdae01353d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0144980 0x7fdae00aaa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00ad010 con 0x7fdae00b4370 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00aca20 con 0x7fdae0144980 2023-12-13T23:57:43.558 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae837d640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae0136370 con 0x7fdae00ceb70 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae6119640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae6119640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41594/0 (socket says 172.21.15.124:41594) 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.556+0000 7fdae6119640 1 -- 172.21.15.124:0/228061135 learned_addr learned my addr 172.21.15.124:0/228061135 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 -- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdae00ceb70 msgr2=0x7fdae01353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdae00ceb70 0x7fdae01353d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 -- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0144980 msgr2=0x7fdae00aaa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0144980 0x7fdae00aaa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 -- 172.21.15.124:0/228061135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc009400 con 0x7fdae00b4370 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae6119640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fdadc000c00 tx=0x7fdadc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.559 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc03aa60 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdadc03abe0 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae0135db0 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdadc0443f0 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.557+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae00acc30 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.558+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae0069400 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.558+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdadc04d450 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.559+0000 7fdad6ffd640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 0x7fdac406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.559+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdadc0abbe0 con 0x7fdae00b4370 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.559+0000 7fdae5918640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 0x7fdac406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.560 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.559+0000 7fdae5918640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 0x7fdac406d680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdad0007fd0 tx=0x7fdad0007920 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.562 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.563+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdadc0769a0 con 0x7fdae00b4370 2023-12-13T23:57:43.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.731+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdae0143f30 con 0x7fdae00b4370 2023-12-13T23:57:43.731 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.732+0000 7fdad6ffd640 1 -- 172.21.15.124:0/228061135 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdadc035070 con 0x7fdae00b4370 2023-12-13T23:57:43.731 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 1 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 msgr2=0x7fdac406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 0x7fdac406d680 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fdad0007fd0 tx=0x7fdad0007920 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 msgr2=0x7fdae0155090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fdadc000c00 tx=0x7fdadc0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 shutdown_connections 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdac406b1c0 0x7fdac406d680 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdae0144980 0x7fdae00aaa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdae00ceb70 0x7fdae01353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 --2- 172.21.15.124:0/228061135 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdae00b4370 0x7fdae0155090 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 >> 172.21.15.124:0/228061135 conn(0x7fdae006a4d0 msgr2=0x7fdae01598c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 shutdown_connections 2023-12-13T23:57:43.734 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.734+0000 7fdae837d640 1 -- 172.21.15.124:0/228061135 wait complete. 2023-12-13T23:57:43.744 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-12-13T23:57:43.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.848+0000 7ff090fa9640 1 Processor -- start 2023-12-13T23:57:43.848 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.848+0000 7ff090fa9640 1 -- start start 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff090fa9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 0x7ff08c0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff090fa9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff08c0d3730 0x7ff08c0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff090fa9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff08c0b2b10 con 0x7ff08c15d330 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff090fa9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0d3bc0 con 0x7ff08c0d3730 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff090fa9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08c0d1590 con 0x7ff08c0d2910 2023-12-13T23:57:43.849 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff08ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff08c0d3730 0x7ff08c0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.849+0000 7ff08ad9d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff08c0d3730 0x7ff08c0b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41608/0 (socket says 172.21.15.124:41608) 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 -- 172.21.15.124:0/3032102393 learned_addr learned my addr 172.21.15.124:0/3032102393 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff089d9b640 1 --2- 172.21.15.124:0/3032102393 >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 0x7ff08c0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 -- 172.21.15.124:0/3032102393 >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 msgr2=0x7ff08c0d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 --2- 172.21.15.124:0/3032102393 >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 0x7ff08c0d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 -- 172.21.15.124:0/3032102393 >> v1:172.21.15.203:6789/0 conn(0x7ff08c15d330 legacy=0x7ff08c0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 -- 172.21.15.124:0/3032102393 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c0d1870 con 0x7ff08c0d3730 2023-12-13T23:57:43.850 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff089d9b640 1 --2- 172.21.15.124:0/3032102393 >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 0x7ff08c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:43.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.850+0000 7ff08ad9d640 1 --2- 172.21.15.124:0/3032102393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c0b6390 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0d1540 tx=0x7ff08402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e996037849272a6c server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff08959a640 1 -- 172.21.15.124:0/3032102393 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0840317e0 con 0x7ff08c0d3730 2023-12-13T23:57:43.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff08959a640 1 -- 172.21.15.124:0/3032102393 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff084031940 con 0x7ff08c0d3730 2023-12-13T23:57:43.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff08959a640 1 -- 172.21.15.124:0/3032102393 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff084031cd0 con 0x7ff08c0d3730 2023-12-13T23:57:43.851 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 -- 172.21.15.124:0/3032102393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 msgr2=0x7ff08c0b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 --2- 172.21.15.124:0/3032102393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c0b6390 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0d1540 tx=0x7ff08402b880 comp rx=0 tx=0).stop 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 -- 172.21.15.124:0/3032102393 shutdown_connections 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 --2- 172.21.15.124:0/3032102393 >> v2:172.21.15.203:3300/0 conn(0x7ff08c0d2910 0x7ff08c0d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 --2- 172.21.15.124:0/3032102393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c0b6390 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 -- 172.21.15.124:0/3032102393 >> 172.21.15.124:0/3032102393 conn(0x7ff08c072020 msgr2=0x7ff08c072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 -- 172.21.15.124:0/3032102393 shutdown_connections 2023-12-13T23:57:43.852 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.851+0000 7ff090fa9640 1 -- 172.21.15.124:0/3032102393 wait complete. 2023-12-13T23:57:43.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 Processor -- start 2023-12-13T23:57:43.854 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 -- start start 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 0x7ff08c0c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 0x7ff08c0b8450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c7f70 con 0x7ff08c0d3730 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c70e0 con 0x7ff08c15d330 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff090fa9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08c0c72f0 con 0x7ff08c0d2910 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff089d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff089d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41610/0 (socket says 172.21.15.124:41610) 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.853+0000 7ff089d9b640 1 -- 172.21.15.124:0/726640770 learned_addr learned my addr 172.21.15.124:0/726640770 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff08ad9d640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 0x7ff08c0b8450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff08a59c640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 0x7ff08c0c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 -- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 msgr2=0x7ff08c0c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 0x7ff08c0c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.855 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 -- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 msgr2=0x7ff08c0b8450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 0x7ff08c0b8450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 -- 172.21.15.124:0/726640770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0840094d0 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff08a59c640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 0x7ff08c0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff08ad9d640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 0x7ff08c0b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff089d9b640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff08000e7e0 tx=0x7ff08000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080004030 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080004190 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff08c0c75d0 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff080004520 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.854+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff08c0c7b90 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.855+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff08c0c81e0 con 0x7ff08c0d3730 2023-12-13T23:57:43.856 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.855+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff08000c6d0 con 0x7ff08c0d3730 2023-12-13T23:57:43.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.856+0000 7ff07affd640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 0x7ff06c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:43.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.856+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0800896e0 con 0x7ff08c0d3730 2023-12-13T23:57:43.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.856+0000 7ff08a59c640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 0x7ff06c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:43.857 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.856+0000 7ff08a59c640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 0x7ff06c06d680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009fc0 tx=0x7ff07c009550 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:43.859 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:43.860+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff080054410 con 0x7ff08c0d3730 2023-12-13T23:57:44.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.024+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff08c069400 con 0x7ff08c0d3730 2023-12-13T23:57:44.024 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.025+0000 7ff07affd640 1 -- 172.21.15.124:0/726640770 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v50) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff080057a30 con 0x7ff08c0d3730 2023-12-13T23:57:44.024 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 32 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.026+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 msgr2=0x7ff06c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.026+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 0x7ff06c06d680 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7ff07c009fc0 tx=0x7ff07c009550 comp rx=0 tx=0).stop 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.026+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 msgr2=0x7ff08c1547f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.026+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7ff08000e7e0 tx=0x7ff08000eca0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 shutdown_connections 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff06c06b1c0 0x7ff06c06d680 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.026 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff08c15d330 0x7ff08c0b8450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff08c0d3730 0x7ff08c1547f0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 --2- 172.21.15.124:0/726640770 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff08c0d2910 0x7ff08c0c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 >> 172.21.15.124:0/726640770 conn(0x7ff08c072020 msgr2=0x7ff08c0b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 shutdown_connections 2023-12-13T23:57:44.027 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.027+0000 7ff090fa9640 1 -- 172.21.15.124:0/726640770 wait complete. 2023-12-13T23:57:44.037 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-12-13T23:57:44.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 Processor -- start 2023-12-13T23:57:44.139 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 -- start start 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 0x7f944c06b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f944c0ad640 0x7f944c06ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c0aeb80 0x7f944c160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0adae0 con 0x7f944c0ad640 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944c0addc0 con 0x7f944c15d1d0 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.140+0000 7f9450eb4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c0ae0a0 con 0x7f944c0aeb80 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f944c0ad640 0x7f944c06ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f944c0ad640 0x7f944c06ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41618/0 (socket says 172.21.15.124:41618) 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 -- 172.21.15.124:0/3042835941 learned_addr learned my addr 172.21.15.124:0/3042835941 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f944a59c640 1 --2- 172.21.15.124:0/3042835941 >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 0x7f944c06b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f944ad9d640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c0aeb80 0x7f944c160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 -- 172.21.15.124:0/3042835941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c0aeb80 msgr2=0x7f944c160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c0aeb80 0x7f944c160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 -- 172.21.15.124:0/3042835941 >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 msgr2=0x7f944c06b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 --2- 172.21.15.124:0/3042835941 >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 0x7f944c06b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f9441d9b640 1 -- 172.21.15.124:0/3042835941 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944c0ae380 con 0x7f944c0ad640 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.141+0000 7f944a59c640 1 --2- 172.21.15.124:0/3042835941 >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 0x7f944c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9441d9b640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0ad640 0x7f944c06ba20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f944c072310 tx=0x7f943402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25278536f698504f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9449d9b640 1 -- 172.21.15.124:0/3042835941 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434031870 con 0x7f944c0ad640 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9449d9b640 1 -- 172.21.15.124:0/3042835941 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94340319d0 con 0x7f944c0ad640 2023-12-13T23:57:44.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9449d9b640 1 -- 172.21.15.124:0/3042835941 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9434031d60 con 0x7f944c0ad640 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 -- 172.21.15.124:0/3042835941 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0ad640 msgr2=0x7f944c06ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0ad640 0x7f944c06ba20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f944c072310 tx=0x7f943402b910 comp rx=0 tx=0).stop 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 -- 172.21.15.124:0/3042835941 shutdown_connections 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0ad640 0x7f944c06ba20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 --2- 172.21.15.124:0/3042835941 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c0aeb80 0x7f944c160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.145 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 --2- 172.21.15.124:0/3042835941 >> v2:172.21.15.203:3300/0 conn(0x7f944c15d1d0 0x7f944c06b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 -- 172.21.15.124:0/3042835941 >> 172.21.15.124:0/3042835941 conn(0x7f944c070e70 msgr2=0x7f944c071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 -- 172.21.15.124:0/3042835941 shutdown_connections 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.142+0000 7f9450eb4640 1 -- 172.21.15.124:0/3042835941 wait complete. 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 Processor -- start 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 -- start start 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f944c0ad640 0x7f944c152400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c15d1d0 0x7f944c0cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c15d9e0 con 0x7f944c0aeb80 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944c149860 con 0x7f944c0ad640 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.144+0000 7f9450eb4640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944c155d70 con 0x7f944c15d1d0 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41624/0 (socket says 172.21.15.124:41624) 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 -- 172.21.15.124:0/2832780365 learned_addr learned my addr 172.21.15.124:0/2832780365 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:44.146 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f944a59c640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f944c0ad640 0x7f944c152400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f944ad9d640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c15d1d0 0x7f944c0cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 -- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c15d1d0 msgr2=0x7f944c0cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c15d1d0 0x7f944c0cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 -- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f944c0ad640 msgr2=0x7f944c152400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f944c0ad640 0x7f944c152400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 -- 172.21.15.124:0/2832780365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94340094d0 con 0x7f944c0aeb80 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9441d9b640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f943402b950 tx=0x7f94340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943403ad30 con 0x7f944c0aeb80 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94340443f0 con 0x7f944c0aeb80 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944c156030 con 0x7f944c0aeb80 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94340447a0 con 0x7f944c0aeb80 2023-12-13T23:57:44.147 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.145+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944c1565f0 con 0x7f944c0aeb80 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.147+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944c15db50 con 0x7f944c0aeb80 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.147+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f943404d450 con 0x7f944c0aeb80 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.147+0000 7f94437fe640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 0x7f942c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.147+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f943402d040 con 0x7f944c0aeb80 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.148+0000 7f944a59c640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 0x7f942c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.148 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.148+0000 7f944a59c640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 0x7f942c06d680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f943c006fd0 tx=0x7f943c008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.151 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.152+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f9434076d60 con 0x7f944c0aeb80 2023-12-13T23:57:44.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.318+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f944c069400 con 0x7f944c0aeb80 2023-12-13T23:57:44.317 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.318+0000 7f94437fe640 1 -- 172.21.15.124:0/2832780365 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v50) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9434035070 con 0x7f944c0aeb80 2023-12-13T23:57:44.318 INFO:teuthology.orchestra.run.smithi124.stdout:pg_num: 64 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 msgr2=0x7f942c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 0x7f942c06d680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f943c006fd0 tx=0x7f943c008040 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 msgr2=0x7f944c152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f943402b950 tx=0x7f94340046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 shutdown_connections 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f942c06b1c0 0x7f942c06d680 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f944c15d1d0 0x7f944c0cd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f944c0aeb80 0x7f944c152940 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 --2- 172.21.15.124:0/2832780365 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f944c0ad640 0x7f944c152400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 >> 172.21.15.124:0/2832780365 conn(0x7f944c070e70 msgr2=0x7f944c06d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 shutdown_connections 2023-12-13T23:57:44.320 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.320+0000 7f9450eb4640 1 -- 172.21.15.124:0/2832780365 wait complete. 2023-12-13T23:57:44.330 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:44.431 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.432+0000 7f7bab094640 1 Processor -- start 2023-12-13T23:57:44.432 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.432+0000 7f7bab094640 1 -- start start 2023-12-13T23:57:44.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 0x7f7ba406b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ba40ad640 0x7f7ba406ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7ba40aeb80 0x7f7ba4160ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba40adae0 con 0x7f7ba40ad640 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba40addc0 con 0x7f7ba40aeb80 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7bab094640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40ae0a0 con 0x7f7ba415d1d0 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ba40ad640 0x7f7ba406ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ba40ad640 0x7f7ba406ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41632/0 (socket says 172.21.15.124:41632) 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 -- 172.21.15.124:0/765674167 learned_addr learned my addr 172.21.15.124:0/765674167 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:44.437 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba9631640 1 --2- 172.21.15.124:0/765674167 >> v2:172.21.15.203:3300/0 conn(0x7f7ba40aeb80 0x7f7ba4160ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba8e30640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 0x7f7ba406b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 -- 172.21.15.124:0/765674167 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 msgr2=0x7f7ba406b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 0x7f7ba406b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 -- 172.21.15.124:0/765674167 >> v2:172.21.15.203:3300/0 conn(0x7f7ba40aeb80 msgr2=0x7f7ba4160ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 --2- 172.21.15.124:0/765674167 >> v2:172.21.15.203:3300/0 conn(0x7f7ba40aeb80 0x7f7ba4160ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.438 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba3fff640 1 -- 172.21.15.124:0/765674167 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba40ae380 con 0x7f7ba40ad640 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.433+0000 7f7ba8e30640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 0x7f7ba406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.434+0000 7f7ba3fff640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba40ad640 0x7f7ba406ba20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7ba406f410 tx=0x7f7b9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f7129009eec892 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.434+0000 7f7ba37fe640 1 -- 172.21.15.124:0/765674167 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b90031870 con 0x7f7ba40ad640 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.434+0000 7f7ba37fe640 1 -- 172.21.15.124:0/765674167 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b900319d0 con 0x7f7ba40ad640 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.434+0000 7f7ba37fe640 1 -- 172.21.15.124:0/765674167 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b90031d60 con 0x7f7ba40ad640 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 -- 172.21.15.124:0/765674167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba40ad640 msgr2=0x7f7ba406ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.439 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba40ad640 0x7f7ba406ba20 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f7ba406f410 tx=0x7f7b9002b910 comp rx=0 tx=0).stop 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 -- 172.21.15.124:0/765674167 shutdown_connections 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba40ad640 0x7f7ba406ba20 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 --2- 172.21.15.124:0/765674167 >> v2:172.21.15.203:3300/0 conn(0x7f7ba40aeb80 0x7f7ba4160ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 --2- 172.21.15.124:0/765674167 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba415d1d0 0x7f7ba406b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 -- 172.21.15.124:0/765674167 >> 172.21.15.124:0/765674167 conn(0x7f7ba4070e70 msgr2=0x7f7ba4071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 -- 172.21.15.124:0/765674167 shutdown_connections 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.435+0000 7f7bab094640 1 -- 172.21.15.124:0/765674167 wait complete. 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.436+0000 7f7bab094640 1 Processor -- start 2023-12-13T23:57:44.440 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.436+0000 7f7bab094640 1 -- start start 2023-12-13T23:57:44.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba40ad640 0x7f7ba41f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ba40aeb80 0x7f7ba41efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba41f0300 con 0x7f7ba415d1d0 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40cea60 con 0x7f7ba40aeb80 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7bab094640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba40ced40 con 0x7f7ba40ad640 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41634/0 (socket says 172.21.15.124:41634) 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 -- 172.21.15.124:0/158858327 learned_addr learned my addr 172.21.15.124:0/158858327 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba8e30640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba40ad640 0x7f7ba41f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.446 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba3fff640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ba40aeb80 0x7f7ba41efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 -- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba40ad640 msgr2=0x7f7ba41f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba40ad640 0x7f7ba41f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 -- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ba40aeb80 msgr2=0x7f7ba41efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ba40aeb80 0x7f7ba41efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 -- 172.21.15.124:0/158858327 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b900094d0 con 0x7f7ba415d1d0 2023-12-13T23:57:44.447 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.437+0000 7f7ba9631640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c007e60 tx=0x7f7b9c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.438+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c014620 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.438+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b9c00ee90 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.438+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba40cf020 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.438+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b9c0149e0 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.438+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba40ce730 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.439+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba4069400 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.439+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7b9c014b40 con 0x7f7ba415d1d0 2023-12-13T23:57:44.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.439+0000 7f7ba17fa640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 0x7f7b8806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.440+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7b9c08a140 con 0x7f7ba415d1d0 2023-12-13T23:57:44.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.440+0000 7f7ba8e30640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 0x7f7b8806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.440+0000 7f7ba8e30640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 0x7f7b8806d680 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800aac0 tx=0x7f7b98008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.449 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.444+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7b9c054fb0 con 0x7f7ba415d1d0 2023-12-13T23:57:44.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.643+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ba406b610 con 0x7f7ba415d1d0 2023-12-13T23:57:44.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.643+0000 7f7ba17fa640 1 -- 172.21.15.124:0/158858327 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f7b9c0585d0 con 0x7f7ba415d1d0 2023-12-13T23:57:44.643 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:44.643 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:44.643 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 msgr2=0x7f7b8806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 0x7f7b8806d680 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7b9800aac0 tx=0x7f7b98008040 comp rx=0 tx=0).stop 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 msgr2=0x7f7ba40ce1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f7b9c007e60 tx=0x7f7b9c00e4b0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 shutdown_connections 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7b8806b1c0 0x7f7b8806d680 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ba40aeb80 0x7f7ba41efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ba415d1d0 0x7f7ba40ce1f0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 --2- 172.21.15.124:0/158858327 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ba40ad640 0x7f7ba41f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 >> 172.21.15.124:0/158858327 conn(0x7f7ba4070e70 msgr2=0x7f7ba4159380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 shutdown_connections 2023-12-13T23:57:44.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.645+0000 7f7bab094640 1 -- 172.21.15.124:0/158858327 wait complete. 2023-12-13T23:57:44.657 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:44.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.757+0000 7f896fc07640 1 Processor -- start 2023-12-13T23:57:44.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896fc07640 1 -- start start 2023-12-13T23:57:44.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896fc07640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f89680d3900 0x7f89680b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896fc07640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680b2b10 con 0x7f896815d330 2023-12-13T23:57:44.767 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896fc07640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89680d1590 con 0x7f89680d2910 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896fc07640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89680d1870 con 0x7f89680d3900 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896d9a3640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f896815d330 0x7f89680b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34608/0 (socket says 172.21.15.124:34608) 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.758+0000 7f896d9a3640 1 -- 172.21.15.124:0/3628324142 learned_addr learned my addr 172.21.15.124:0/3628324142 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288674623 0 0) 0x7f89680b2b10 con 0x7f896815d330 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896e1a4640 1 --2- 172.21.15.124:0/3628324142 >> v2:172.21.15.203:3300/0 conn(0x7f89680d3900 0x7f89680b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.768 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8954003520 con 0x7f896815d330 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3513475213 0 0) 0x7f8954003520 con 0x7f896815d330 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 >> v2:172.21.15.203:3300/0 conn(0x7f89680d3900 msgr2=0x7f89680b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 --2- 172.21.15.124:0/3628324142 >> v2:172.21.15.203:3300/0 conn(0x7f89680d3900 0x7f89680b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 >> v1:172.21.15.203:6789/0 conn(0x7f89680d2910 legacy=0x7f89680d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680d1b50 con 0x7f896815d330 2023-12-13T23:57:44.769 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f8958001670 con 0x7f896815d330 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 >> v1:172.21.15.124:6789/0 conn(0x7f896815d330 legacy=0x7f89680b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f8954003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89540047c0 0x7f8954004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89540051d0 0x7f895400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89680d1870 con 0x7f89540039c0 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895400c120 con 0x7f89540051d0 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895400c2e0 con 0x7f89540047c0 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f8954003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d1a2640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89540047c0 0x7f8954004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.770 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896e1a4640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89540051d0 0x7f895400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 -- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89540047c0 msgr2=0x7f8954004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89540047c0 0x7f8954004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 -- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89540051d0 msgr2=0x7f895400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89540051d0 0x7f895400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896e1a4640 1 -- 172.21.15.124:0/3628324142 reap_dead start 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.759+0000 7f896d9a3640 1 -- 172.21.15.124:0/3628324142 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954003520 con 0x7f89540039c0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.760+0000 7f896d9a3640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f8954003de0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8958001eb0 tx=0x7f89540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aaf734a9eb7221e server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.760+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8958005ad0 con 0x7f89540039c0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.760+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8958005c30 con 0x7f89540039c0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.760+0000 7f896c9a1640 1 -- 172.21.15.124:0/3628324142 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89580072b0 con 0x7f89540039c0 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 -- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 msgr2=0x7f8954003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f8954003de0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f8958001eb0 tx=0x7f89540057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.771 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 -- 172.21.15.124:0/3628324142 shutdown_connections 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 --2- 172.21.15.124:0/3628324142 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f8954003de0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 -- 172.21.15.124:0/3628324142 >> 172.21.15.124:0/3628324142 conn(0x7f8968072020 msgr2=0x7f8968072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 -- 172.21.15.124:0/3628324142 shutdown_connections 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.761+0000 7f896fc07640 1 -- 172.21.15.124:0/3628324142 wait complete. 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.762+0000 7f896fc07640 1 Processor -- start 2023-12-13T23:57:44.772 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.762+0000 7f896fc07640 1 -- start start 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89680b3d70 0x7f89680b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89680ca4a0 0x7f89680ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89681f7090 con 0x7f89540039c0 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89680c7ae0 con 0x7f89680b3d70 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896fc07640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89680b46f0 con 0x7f89680ca4a0 2023-12-13T23:57:44.773 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41652/0 (socket says 172.21.15.124:41652) 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 -- 172.21.15.124:0/2693464920 learned_addr learned my addr 172.21.15.124:0/2693464920 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d1a2640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89680b3d70 0x7f89680b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896e1a4640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89680ca4a0 0x7f89680ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 -- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89680ca4a0 msgr2=0x7f89680ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.774 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89680ca4a0 0x7f89680ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 -- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89680b3d70 msgr2=0x7f89680b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89680b3d70 0x7f89680b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 -- 172.21.15.124:0/2693464920 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89680c6ed0 con 0x7f89540039c0 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f896d9a3640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f8958000c00 tx=0x7f895802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895803c040 con 0x7f89540039c0 2023-12-13T23:57:44.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.763+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89580075e0 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.764+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89680c71b0 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.764+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895802c960 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.764+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89680c7770 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.765+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89680c7900 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.765+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8958032070 con 0x7f89540039c0 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.765+0000 7f895e7fc640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 0x7f894c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:44.776 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.765+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89580aada0 con 0x7f89540039c0 2023-12-13T23:57:44.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.766+0000 7f896d1a2640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 0x7f894c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:44.777 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.766+0000 7f896d1a2640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 0x7f894c06d680 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8964000c00 tx=0x7f8964009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:44.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.769+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8958075b60 con 0x7f89540039c0 2023-12-13T23:57:44.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.936+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8968069400 con 0x7f89540039c0 2023-12-13T23:57:44.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.937+0000 7f895e7fc640 1 -- 172.21.15.124:0/2693464920 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f8958007750 con 0x7f89540039c0 2023-12-13T23:57:44.937 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:44.938 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:44.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 msgr2=0x7f894c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.939 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 0x7f894c06d680 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8964000c00 tx=0x7f8964009040 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 msgr2=0x7f89681f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f8958000c00 tx=0x7f895802e070 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 shutdown_connections 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f894c06b1c0 0x7f894c06d680 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f89680b3d70 0x7f89680b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f89680ca4a0 0x7f89680ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 --2- 172.21.15.124:0/2693464920 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f89540039c0 0x7f89681f6b50 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 >> 172.21.15.124:0/2693464920 conn(0x7f8968072020 msgr2=0x7f896815f340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:44.940 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 shutdown_connections 2023-12-13T23:57:44.941 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:44.939+0000 7f896fc07640 1 -- 172.21.15.124:0/2693464920 wait complete. 2023-12-13T23:57:44.950 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-13T23:57:44.951 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:45.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.051+0000 7fe96aa9b640 1 Processor -- start 2023-12-13T23:57:45.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.051+0000 7fe96aa9b640 1 -- start start 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe96aa9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe964143490 0x7fe9640b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe96aa9b640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96415e2a0 con 0x7fe96415e730 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe96aa9b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96415b240 con 0x7fe96415abf0 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe96aa9b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96415b520 con 0x7fe964143490 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe964143490 0x7fe9640b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe964143490 0x7fe9640b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41656/0 (socket says 172.21.15.124:41656) 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe963fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe96415e730 0x7fe96415a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34612/0 (socket says 172.21.15.124:34612) 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 -- 172.21.15.124:0/1502401120 learned_addr learned my addr 172.21.15.124:0/1502401120 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 -- 172.21.15.124:0/1502401120 >> v1:172.21.15.203:6789/0 conn(0x7fe96415abf0 legacy=0x7fe9640b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 -- 172.21.15.124:0/1502401120 >> v1:172.21.15.124:6789/0 conn(0x7fe96415e730 legacy=0x7fe96415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.249 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.052+0000 7fe968837640 1 -- 172.21.15.124:0/1502401120 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96415dc50 con 0x7fe964143490 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.053+0000 7fe968837640 1 --2- 172.21.15.124:0/1502401120 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe964143490 0x7fe9640b0cf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe95400bb10 tx=0x7fe95402e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dee8322b40d3179 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.053+0000 7fe9637fe640 1 -- 172.21.15.124:0/1502401120 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954036440 con 0x7fe964143490 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.053+0000 7fe9637fe640 1 -- 172.21.15.124:0/1502401120 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9540365a0 con 0x7fe964143490 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.053+0000 7fe9637fe640 1 -- 172.21.15.124:0/1502401120 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe954036930 con 0x7fe964143490 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1502401120 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe964143490 msgr2=0x7fe9640b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1502401120 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe964143490 0x7fe9640b0cf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe95400bb10 tx=0x7fe95402e170 comp rx=0 tx=0).stop 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1502401120 shutdown_connections 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1502401120 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe964143490 0x7fe9640b0cf0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1502401120 >> 172.21.15.124:0/1502401120 conn(0x7fe96406a4d0 msgr2=0x7fe96406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1502401120 shutdown_connections 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.054+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1502401120 wait complete. 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.055+0000 7fe96aa9b640 1 Processor -- start 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.055+0000 7fe96aa9b640 1 -- start start 2023-12-13T23:57:45.250 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe964143490 0x7fe964140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe96415e730 0x7fe964158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96413d9a0 con 0x7fe96415abf0 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe964159e60 con 0x7fe964143490 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe96aa9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe964141080 con 0x7fe96415e730 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41666/0 (socket says 172.21.15.124:41666) 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 -- 172.21.15.124:0/1691627766 learned_addr learned my addr 172.21.15.124:0/1691627766 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 -- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe96415e730 msgr2=0x7fe964158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe96415e730 0x7fe964158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 -- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe964143490 msgr2=0x7fe964140980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe964143490 0x7fe964140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 -- 172.21.15.124:0/1691627766 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95400b770 con 0x7fe96415abf0 2023-12-13T23:57:45.251 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.056+0000 7fe963fff640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe95800c410 tx=0x7fe95800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.057+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580074e0 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.057+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958007640 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.057+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe96413d2e0 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.057+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580049e0 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.057+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe96413e300 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.058+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe964143c70 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.058+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe958004b40 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.058+0000 7fe9617fa640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 0x7fe95006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.058+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe958089310 con 0x7fe96415abf0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.059+0000 7fe968837640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 0x7fe95006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.059+0000 7fe968837640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 0x7fe95006d680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe954038000 tx=0x7fe954030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.252 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.063+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe9580540d0 con 0x7fe96415abf0 2023-12-13T23:57:45.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.263+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe96413dae0 con 0x7fe96415abf0 2023-12-13T23:57:45.262 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.263+0000 7fe9617fa640 1 -- 172.21.15.124:0/1691627766 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7fe96413dae0 con 0x7fe96415abf0 2023-12-13T23:57:45.262 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:45.263 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:45.263 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 msgr2=0x7fe95006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 0x7fe95006d680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe954038000 tx=0x7fe954030000 comp rx=0 tx=0).stop 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 msgr2=0x7fe96413ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fe95800c410 tx=0x7fe95800c8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 shutdown_connections 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe95006b1c0 0x7fe95006d680 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe96415e730 0x7fe964158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.265 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe96415abf0 0x7fe96413ddc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 --2- 172.21.15.124:0/1691627766 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe964143490 0x7fe964140980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 >> 172.21.15.124:0/1691627766 conn(0x7fe96406a4d0 msgr2=0x7fe96415a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 shutdown_connections 2023-12-13T23:57:45.266 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.265+0000 7fe96aa9b640 1 -- 172.21.15.124:0/1691627766 wait complete. 2023-12-13T23:57:45.275 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:57:45.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.376+0000 7f637b55a640 1 Processor -- start 2023-12-13T23:57:45.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.376+0000 7f637b55a640 1 -- start start 2023-12-13T23:57:45.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f637b55a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63740b4370 0x7f63740b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f637b55a640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6374071770 con 0x7f63740ceb70 2023-12-13T23:57:45.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f637b55a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63740ccb60 con 0x7f6374144980 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f637b55a640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637415ced0 con 0x7f63740b4370 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6379af7640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63740b4370 0x7f63740b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6378af5640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f63740ceb70 0x7f63740cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34628/0 (socket says 172.21.15.124:34628) 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6378af5640 1 -- 172.21.15.124:0/2999148646 learned_addr learned my addr 172.21.15.124:0/2999148646 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6379af7640 1 -- 172.21.15.124:0/2999148646 >> v1:172.21.15.203:6789/0 conn(0x7f6374144980 legacy=0x7f637415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6379af7640 1 -- 172.21.15.124:0/2999148646 >> v1:172.21.15.124:6789/0 conn(0x7f63740ceb70 legacy=0x7f63740cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f6379af7640 1 -- 172.21.15.124:0/2999148646 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63740b4ce0 con 0x7f63740b4370 2023-12-13T23:57:45.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.377+0000 7f636bfff640 1 -- 172.21.15.124:0/2999148646 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933422387 0 0) 0x7f6374071770 con 0x7f63740ceb70 2023-12-13T23:57:45.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.378+0000 7f6379af7640 1 --2- 172.21.15.124:0/2999148646 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740b4370 0x7f63740b47a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f63740b4e70 tx=0x7f636c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86989e48811f3010 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.378+0000 7f636bfff640 1 -- 172.21.15.124:0/2999148646 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c02cc50 con 0x7f63740b4370 2023-12-13T23:57:45.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.378+0000 7f636bfff640 1 -- 172.21.15.124:0/2999148646 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c02cdb0 con 0x7f63740b4370 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.378+0000 7f636bfff640 1 -- 172.21.15.124:0/2999148646 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c034660 con 0x7f63740b4370 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.378+0000 7f637b55a640 1 -- 172.21.15.124:0/2999148646 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740b4370 msgr2=0x7f63740b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 --2- 172.21.15.124:0/2999148646 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740b4370 0x7f63740b47a0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f63740b4e70 tx=0x7f636c02bea0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 -- 172.21.15.124:0/2999148646 shutdown_connections 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 --2- 172.21.15.124:0/2999148646 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740b4370 0x7f63740b47a0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 -- 172.21.15.124:0/2999148646 >> 172.21.15.124:0/2999148646 conn(0x7f637406a4d0 msgr2=0x7f637406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 -- 172.21.15.124:0/2999148646 shutdown_connections 2023-12-13T23:57:45.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.379+0000 7f637b55a640 1 -- 172.21.15.124:0/2999148646 wait complete. 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 Processor -- start 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 -- start start 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63740b4370 0x7f637413dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6374144980 0x7f63740cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.381 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63741f18b0 con 0x7f63740ceb70 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63741f12f0 con 0x7f6374144980 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.380+0000 7f637b55a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63741f15d0 con 0x7f63740b4370 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41678/0 (socket says 172.21.15.124:41678) 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 -- 172.21.15.124:0/1417272876 learned_addr learned my addr 172.21.15.124:0/1417272876 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f63792f6640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63740b4370 0x7f637413dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6379af7640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6374144980 0x7f63740cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 -- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63740b4370 msgr2=0x7f637413dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63740b4370 0x7f637413dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 -- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6374144980 msgr2=0x7f63740cbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6374144980 0x7f63740cbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 -- 172.21.15.124:0/1417272876 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c009a70 con 0x7f63740ceb70 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6378af5640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f636400c4d0 tx=0x7f636400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63640074e0 con 0x7f63740ceb70 2023-12-13T23:57:45.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6364007660 con 0x7f63740ceb70 2023-12-13T23:57:45.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63740aff60 con 0x7f63740ceb70 2023-12-13T23:57:45.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6364004a20 con 0x7f63740ceb70 2023-12-13T23:57:45.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.381+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6374141350 con 0x7f63740ceb70 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.382+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63740b0480 con 0x7f63740ceb70 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.383+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6364004b80 con 0x7f63740ceb70 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.383+0000 7f6369ffb640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 0x7f635806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.383+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6364088da0 con 0x7f63740ceb70 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.383+0000 7f63792f6640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 0x7f635806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.384+0000 7f63792f6640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 0x7f635806d6b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f63741430f0 tx=0x7f6370009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.387+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f63740b0480 con 0x7f63740ceb70 2023-12-13T23:57:45.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.590+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63740b0480 con 0x7f63740ceb70 2023-12-13T23:57:45.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.590+0000 7f6369ffb640 1 -- 172.21.15.124:0/1417272876 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+4925 (secure 0 0 0) 0x7f63740b0480 con 0x7f63740ceb70 2023-12-13T23:57:45.590 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:45.590 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":22,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-12-13T23:57:17.326453+0000","modified":"2023-12-13T23:57:19.327808+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5910},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5910":{"gid":5910,"name":"b","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.203:6837/2421641365","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":2421641365},{"type":"v1","addr":"172.21.15.203:6837","nonce":2421641365}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-12-13T23:57:45.590 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 22 2023-12-13T23:57:45.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 msgr2=0x7f635806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 0x7f635806d6b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f63741430f0 tx=0x7f6370009060 comp rx=0 tx=0).stop 2023-12-13T23:57:45.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 msgr2=0x7f63741f08c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.592 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f636400c4d0 tx=0x7f636400c990 comp rx=0 tx=0).stop 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 shutdown_connections 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f635806b1f0 0x7f635806d6b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6374144980 0x7f63740cbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63740ceb70 0x7f63741f08c0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 --2- 172.21.15.124:0/1417272876 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f63740b4370 0x7f637413dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 >> 172.21.15.124:0/1417272876 conn(0x7f637406a4d0 msgr2=0x7f637406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 shutdown_connections 2023-12-13T23:57:45.593 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.592+0000 7f637b55a640 1 -- 172.21.15.124:0/1417272876 wait complete. 2023-12-13T23:57:45.603 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:45.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.703+0000 7f8a473fc640 1 Processor -- start 2023-12-13T23:57:45.703 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.703+0000 7f8a473fc640 1 -- start start 2023-12-13T23:57:45.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a473fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 0x7f8a400b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a473fc640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 0x7f8a400b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a473fc640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a400b2b10 con 0x7f8a400d2910 2023-12-13T23:57:45.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a473fc640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a400d1590 con 0x7f8a400d2d90 2023-12-13T23:57:45.704 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a473fc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400d1870 con 0x7f8a4015d330 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a44997640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8a400d2910 0x7f8a400d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34638/0 (socket says 172.21.15.124:34638) 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.704+0000 7f8a44997640 1 -- 172.21.15.124:0/437861533 learned_addr learned my addr 172.21.15.124:0/437861533 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45999640 1 --2- 172.21.15.124:0/437861533 >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 0x7f8a400b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45198640 1 --2- 172.21.15.124:0/437861533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 0x7f8a400b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a37fff640 1 -- 172.21.15.124:0/437861533 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522052250 0 0) 0x7f8a400b2b10 con 0x7f8a400d2910 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a37fff640 1 -- 172.21.15.124:0/437861533 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c003520 con 0x7f8a400d2910 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45198640 1 -- 172.21.15.124:0/437861533 >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 msgr2=0x7f8a400b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45198640 1 --2- 172.21.15.124:0/437861533 >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 0x7f8a400b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45198640 1 -- 172.21.15.124:0/437861533 >> v1:172.21.15.124:6789/0 conn(0x7f8a400d2910 legacy=0x7f8a400d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45198640 1 -- 172.21.15.124:0/437861533 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a400d1b50 con 0x7f8a4015d330 2023-12-13T23:57:45.705 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.705+0000 7f8a45999640 1 --2- 172.21.15.124:0/437861533 >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 0x7f8a400b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:45.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a45198640 1 --2- 172.21.15.124:0/437861533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 0x7f8a400b8b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d90 tx=0x7f8a3c02be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a52a11f24824aa server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.706 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a37fff640 1 -- 172.21.15.124:0/437861533 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c02cbc0 con 0x7f8a4015d330 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a37fff640 1 -- 172.21.15.124:0/437861533 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c02cd20 con 0x7f8a4015d330 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a37fff640 1 -- 172.21.15.124:0/437861533 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c034640 con 0x7f8a4015d330 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 -- 172.21.15.124:0/437861533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 msgr2=0x7f8a400b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 --2- 172.21.15.124:0/437861533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 0x7f8a400b8b80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c009d90 tx=0x7f8a3c02be10 comp rx=0 tx=0).stop 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 -- 172.21.15.124:0/437861533 shutdown_connections 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 --2- 172.21.15.124:0/437861533 >> v2:172.21.15.203:3300/0 conn(0x7f8a400d2d90 0x7f8a400b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 --2- 172.21.15.124:0/437861533 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a4015d330 0x7f8a400b8b80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.706+0000 7f8a473fc640 1 -- 172.21.15.124:0/437861533 >> 172.21.15.124:0/437861533 conn(0x7f8a40072020 msgr2=0x7f8a40072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.707+0000 7f8a473fc640 1 -- 172.21.15.124:0/437861533 shutdown_connections 2023-12-13T23:57:45.707 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.707+0000 7f8a473fc640 1 -- 172.21.15.124:0/437861533 wait complete. 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 Processor -- start 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 -- start start 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a400d2910 0x7f8a401f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 0x7f8a400cc070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400cbce0 con 0x7f8a4015d330 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400cba00 con 0x7f8a400d2910 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.708+0000 7f8a473fc640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a400c6c40 con 0x7f8a400d2d90 2023-12-13T23:57:45.710 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41684/0 (socket says 172.21.15.124:41684) 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 -- 172.21.15.124:0/350518634 learned_addr learned my addr 172.21.15.124:0/350518634 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45198640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a400d2910 0x7f8a401f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a44997640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 0x7f8a400cc070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 -- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 msgr2=0x7f8a400cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 0x7f8a400cc070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 -- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a400d2910 msgr2=0x7f8a401f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a400d2910 0x7f8a401f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 -- 172.21.15.124:0/350518634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38009400 con 0x7f8a4015d330 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a44997640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 0x7f8a400cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a45999640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800e900 tx=0x7f8a3800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a380096e0 con 0x7f8a4015d330 2023-12-13T23:57:45.711 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3800ca00 con 0x7f8a4015d330 2023-12-13T23:57:45.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a3c009a70 con 0x7f8a4015d330 2023-12-13T23:57:45.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3800cd70 con 0x7f8a4015d330 2023-12-13T23:57:45.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.709+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a400c7200 con 0x7f8a4015d330 2023-12-13T23:57:45.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.710+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a400c7420 con 0x7f8a4015d330 2023-12-13T23:57:45.712 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.711+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8a38013470 con 0x7f8a4015d330 2023-12-13T23:57:45.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.711+0000 7f8a35ffb640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 0x7f8a2406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:45.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.711+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a38089c10 con 0x7f8a4015d330 2023-12-13T23:57:45.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.711+0000 7f8a45198640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 0x7f8a2406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:45.713 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.712+0000 7f8a45198640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 0x7f8a2406d680 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c030000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:45.714 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.715+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8a38054a00 con 0x7f8a4015d330 2023-12-13T23:57:45.883 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.884+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a40069400 con 0x7f8a4015d330 2023-12-13T23:57:45.884 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.885+0000 7f8a35ffb640 1 -- 172.21.15.124:0/350518634 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f8a38058020 con 0x7f8a4015d330 2023-12-13T23:57:45.885 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:45.885 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":50,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:18.754782+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2023-12-13T23:57:15.517529+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"49","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2023-12-13T23:57:16.303275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:45.887 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 msgr2=0x7f8a2406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 0x7f8a2406d680 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c036000 tx=0x7f8a3c030000 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 msgr2=0x7f8a400cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800e900 tx=0x7f8a3800edc0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 shutdown_connections 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8a2406b1c0 0x7f8a2406d680 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8a4015d330 0x7f8a400cc5b0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8a400d2d90 0x7f8a400cc070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 --2- 172.21.15.124:0/350518634 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8a400d2910 0x7f8a401f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 >> 172.21.15.124:0/350518634 conn(0x7f8a40072020 msgr2=0x7f8a400b67e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:45.888 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 shutdown_connections 2023-12-13T23:57:45.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:45.887+0000 7f8a473fc640 1 -- 172.21.15.124:0/350518634 wait complete. 2023-12-13T23:57:45.898 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-12-13T23:57:46.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.000+0000 7fe4faf9e640 1 Processor -- start 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.000+0000 7fe4faf9e640 1 -- start start 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4faf9e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe4f40afc60 0x7fe4f40d3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4faf9e640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4f41616d0 0x7fe4f40cf8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4faf9e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f40ce6d0 con 0x7fe4f415e720 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4faf9e640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f40cf460 con 0x7fe4f41616d0 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4faf9e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f40ce920 con 0x7fe4f40afc60 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4f41616d0 0x7fe4f40cf8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe4f41616d0 0x7fe4f40cf8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41698/0 (socket says 172.21.15.124:41698) 2023-12-13T23:57:46.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 -- 172.21.15.124:0/2989059186 learned_addr learned my addr 172.21.15.124:0/2989059186 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4f979b640 1 --2- 172.21.15.124:0/2989059186 >> v2:172.21.15.203:3300/0 conn(0x7fe4f40afc60 0x7fe4f40d3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 -- 172.21.15.124:0/2989059186 >> v2:172.21.15.203:3300/0 conn(0x7fe4f40afc60 msgr2=0x7fe4f40d3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 --2- 172.21.15.124:0/2989059186 >> v2:172.21.15.203:3300/0 conn(0x7fe4f40afc60 0x7fe4f40d3ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 -- 172.21.15.124:0/2989059186 >> v1:172.21.15.203:6789/0 conn(0x7fe4f415e720 legacy=0x7fe4f4160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.001+0000 7fe4fa79d640 1 -- 172.21.15.124:0/2989059186 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f40cec00 con 0x7fe4f41616d0 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.002+0000 7fe4fa79d640 1 --2- 172.21.15.124:0/2989059186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f41616d0 0x7fe4f40cf8f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe4f406afe0 tx=0x7fe4ec02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bf27e51eabe4116 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.002+0000 7fe4f8f9a640 1 -- 172.21.15.124:0/2989059186 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec02c9f0 con 0x7fe4f41616d0 2023-12-13T23:57:46.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.002+0000 7fe4f8f9a640 1 -- 172.21.15.124:0/2989059186 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec02cb50 con 0x7fe4f41616d0 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.002+0000 7fe4f8f9a640 1 -- 172.21.15.124:0/2989059186 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec0343f0 con 0x7fe4f41616d0 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 -- 172.21.15.124:0/2989059186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f41616d0 msgr2=0x7fe4f40cf8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/2989059186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f41616d0 0x7fe4f40cf8f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe4f406afe0 tx=0x7fe4ec02be50 comp rx=0 tx=0).stop 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 -- 172.21.15.124:0/2989059186 shutdown_connections 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/2989059186 >> v2:172.21.15.203:3300/0 conn(0x7fe4f40afc60 0x7fe4f40d3ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/2989059186 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f41616d0 0x7fe4f40cf8f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 -- 172.21.15.124:0/2989059186 >> 172.21.15.124:0/2989059186 conn(0x7fe4f40d2fb0 msgr2=0x7fe4f40d33c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 -- 172.21.15.124:0/2989059186 shutdown_connections 2023-12-13T23:57:46.010 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.003+0000 7fe4faf9e640 1 -- 172.21.15.124:0/2989059186 wait complete. 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.004+0000 7fe4faf9e640 1 Processor -- start 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.004+0000 7fe4faf9e640 1 -- start start 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 0x7fe4f40cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4f41616d0 0x7fe4f40cc210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f415d920 con 0x7fe4f40afc60 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40b6820 con 0x7fe4f415e720 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4faf9e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f40b6b00 con 0x7fe4f41616d0 2023-12-13T23:57:46.011 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41704/0 (socket says 172.21.15.124:41704) 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 -- 172.21.15.124:0/3739843919 learned_addr learned my addr 172.21.15.124:0/3739843919 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 -- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4f41616d0 msgr2=0x7fe4f40cc210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4f41616d0 0x7fe4f40cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 -- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 msgr2=0x7fe4f40cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f979b640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 0x7fe4f40cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 0x7fe4f40cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.012 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 -- 172.21.15.124:0/3739843919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ec009b30 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f979b640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 0x7fe4f40cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4f9f9c640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000b3f0 tx=0x7fe4f000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.005+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0011830 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.006+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4f0011990 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.006+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f40b6de0 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.006+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4f0011d40 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.006+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f40b73a0 con 0x7fe4f40afc60 2023-12-13T23:57:46.013 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.006+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f40b1fa0 con 0x7fe4f40afc60 2023-12-13T23:57:46.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.007+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe4f001a400 con 0x7fe4f40afc60 2023-12-13T23:57:46.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.007+0000 7fe4ea7fc640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 0x7fe4d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.008+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe4f0089860 con 0x7fe4f40afc60 2023-12-13T23:57:46.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.008+0000 7fe4f979b640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 0x7fe4d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.014 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.008+0000 7fe4f979b640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 0x7fe4d806d680 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4006fd0 tx=0x7fe4e4008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.015 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.012+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe4f40b1fa0 con 0x7fe4f40afc60 2023-12-13T23:57:46.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.216+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe4f40b1fa0 con 0x7fe4f40afc60 2023-12-13T23:57:46.216 INFO:tasks.ceph.mon.a.smithi124.stderr:2023-12-13T23:57:46.217+0000 7ffbf0477640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-12-13T23:57:46.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.220+0000 7fe4ea7fc640 1 -- 172.21.15.124:0/3739843919 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v23) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe4f40b1fa0 con 0x7fe4f40afc60 2023-12-13T23:57:46.219 INFO:teuthology.orchestra.run.smithi124.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 msgr2=0x7fe4d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 0x7fe4d806d680 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4e4006fd0 tx=0x7fe4e4008040 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 msgr2=0x7fe4f415ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe4f000b3f0 tx=0x7fe4f000b8b0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 shutdown_connections 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe4d806b1c0 0x7fe4d806d680 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe4f415e720 0x7fe4f40cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe4f41616d0 0x7fe4f40cc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 --2- 172.21.15.124:0/3739843919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe4f40afc60 0x7fe4f415ce50 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 >> 172.21.15.124:0/3739843919 conn(0x7fe4f40d2fb0 msgr2=0x7fe4f4160b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:46.222 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 shutdown_connections 2023-12-13T23:57:46.223 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.222+0000 7fe4faf9e640 1 -- 172.21.15.124:0/3739843919 wait complete. 2023-12-13T23:57:46.224 INFO:tasks.ceph.mds.b.smithi203.stderr: -2609> 2023-12-13T23:57:19.960+0000 7f61035c6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-12-13T23:57:46.233 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-12-13T23:57:46.252 INFO:tasks.ceph.mds.b.smithi203.stdout:starting mds.b at 2023-12-13T23:57:46.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.336+0000 7f7ec407c640 1 Processor -- start 2023-12-13T23:57:46.335 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.336+0000 7f7ec407c640 1 -- start start 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec407c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ebc155790 0x7f7ebc06f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec407c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 0x7f7ebc15ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec407c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc155b70 con 0x7f7ebc15b5e0 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec407c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc06ee30 con 0x7f7ebc155790 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec407c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc06f3b0 con 0x7f7ebc06abe0 2023-12-13T23:57:46.337 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ebc155790 0x7f7ebc06f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ebc155790 0x7f7ebc06f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41712/0 (socket says 172.21.15.124:41712) 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 -- 172.21.15.124:0/2511706656 learned_addr learned my addr 172.21.15.124:0/2511706656 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec2619640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 0x7f7ebc15ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 -- 172.21.15.124:0/2511706656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 msgr2=0x7f7ebc15ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 0x7f7ebc15ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 -- 172.21.15.124:0/2511706656 >> v1:172.21.15.203:6789/0 conn(0x7f7ebc15b5e0 legacy=0x7f7ebc06a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec1e18640 1 -- 172.21.15.124:0/2511706656 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc15c170 con 0x7f7ebc155790 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.337+0000 7f7ec2619640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 0x7f7ebc15ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec1e18640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc155790 0x7f7ebc06f840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c11245a3a8ee224b server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec0e16640 1 -- 172.21.15.124:0/2511706656 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8031640 con 0x7f7ebc155790 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec0e16640 1 -- 172.21.15.124:0/2511706656 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb80317a0 con 0x7f7ebc155790 2023-12-13T23:57:46.338 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec0e16640 1 -- 172.21.15.124:0/2511706656 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8031b30 con 0x7f7ebc155790 2023-12-13T23:57:46.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 -- 172.21.15.124:0/2511706656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc155790 msgr2=0x7f7ebc06f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc155790 0x7f7ebc06f840 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb802b8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 -- 172.21.15.124:0/2511706656 shutdown_connections 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc06abe0 0x7f7ebc15ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 --2- 172.21.15.124:0/2511706656 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc155790 0x7f7ebc06f840 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.338+0000 7f7ec407c640 1 -- 172.21.15.124:0/2511706656 >> 172.21.15.124:0/2511706656 conn(0x7f7ebc0b1e50 msgr2=0x7f7ebc0b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.339+0000 7f7ec407c640 1 -- 172.21.15.124:0/2511706656 shutdown_connections 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.339+0000 7f7ec407c640 1 -- 172.21.15.124:0/2511706656 wait complete. 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 Processor -- start 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 -- start start 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc155790 0x7f7ebc0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ebc15b5e0 0x7f7ebc0b3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc155f60 con 0x7f7ebc06abe0 2023-12-13T23:57:46.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0af610 con 0x7f7ebc15b5e0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec407c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0af8f0 con 0x7f7ebc155790 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.340+0000 7f7ec1e18640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41724/0 (socket says 172.21.15.124:41724) 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 -- 172.21.15.124:0/3768041268 learned_addr learned my addr 172.21.15.124:0/3768041268 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec2619640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ebc15b5e0 0x7f7ebc0b3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 -- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc155790 msgr2=0x7f7ebc0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1617640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc155790 0x7f7ebc0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc155790 0x7f7ebc0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 -- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ebc15b5e0 msgr2=0x7f7ebc0b3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ebc15b5e0 0x7f7ebc0b3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 -- 172.21.15.124:0/3768041268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb80094d0 con 0x7f7ebc06abe0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec1e18640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb803ac60 con 0x7f7ebc06abe0 2023-12-13T23:57:46.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb803ade0 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc1561b0 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8044630 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.341+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc151a70 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.342+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc069400 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.342+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7eb802f050 con 0x7f7ebc06abe0 2023-12-13T23:57:46.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.343+0000 7f7eb27fc640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 0x7f7ea006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:46.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.343+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7eb80abb10 con 0x7f7ebc06abe0 2023-12-13T23:57:46.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.343+0000 7f7ec1617640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 0x7f7ea006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:46.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.343+0000 7f7ec1617640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 0x7f7ea006d680 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc158060 tx=0x7f7eac008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:46.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.347+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7eb8076930 con 0x7f7ebc06abe0 2023-12-13T23:57:46.548 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:46.548+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7ebc0b2c50 con 0x7f7ebc06abe0 2023-12-13T23:57:47.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.221+0000 7f7eb27fc640 1 -- 172.21.15.124:0/3768041268 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v24) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7eb8079f50 con 0x7f7ebc06abe0 2023-12-13T23:57:47.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 msgr2=0x7f7ea006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 0x7f7ea006d680 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc158060 tx=0x7f7eac008040 comp rx=0 tx=0).stop 2023-12-13T23:57:47.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 msgr2=0x7f7ebc151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f7eb8009870 tx=0x7f7eb80046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 shutdown_connections 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7ea006b1c0 0x7f7ea006d680 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ebc15b5e0 0x7f7ebc0b3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ebc155790 0x7f7ebc0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 --2- 172.21.15.124:0/3768041268 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ebc06abe0 0x7f7ebc151530 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 >> 172.21.15.124:0/3768041268 conn(0x7f7ebc0b1e50 msgr2=0x7f7ebc0cdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 shutdown_connections 2023-12-13T23:57:47.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.223+0000 7f7ec407c640 1 -- 172.21.15.124:0/3768041268 wait complete. 2023-12-13T23:57:47.234 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-12-13T23:57:47.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.339+0000 7f8da27b0640 1 Processor -- start 2023-12-13T23:57:47.339 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.339+0000 7f8da27b0640 1 -- start start 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da27b0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8d9c146aa0 0x7f8d9c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da27b0640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0b0cf0 0x7f8d9c0b1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da27b0640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d9c161bb0 con 0x7f8d9c0ac8e0 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da27b0640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9c0b16c0 con 0x7f8d9c146aa0 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da27b0640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d9c160fe0 con 0x7f8d9c0b0cf0 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8d9bfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8d9c146aa0 0x7f8d9c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da0d4d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f8d9c0ac8e0 0x7f8d9c0acd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34646/0 (socket says 172.21.15.124:34646) 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8da0d4d640 1 -- 172.21.15.124:0/108897555 learned_addr learned my addr 172.21.15.124:0/108897555 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8d9bfff640 1 -- 172.21.15.124:0/108897555 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0b0cf0 msgr2=0x7f8d9c0b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:47.340 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8d9bfff640 1 --2- 172.21.15.124:0/108897555 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0b0cf0 0x7f8d9c0b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8d9bfff640 1 -- 172.21.15.124:0/108897555 >> v1:172.21.15.124:6789/0 conn(0x7f8d9c0ac8e0 legacy=0x7f8d9c0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.341 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.340+0000 7f8d9bfff640 1 -- 172.21.15.124:0/108897555 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d9c146e80 con 0x7f8d9c146aa0 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.341+0000 7f8d9bfff640 1 --2- 172.21.15.124:0/108897555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c0b2140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0aeb20 tx=0x7f8d8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40aa4983d4d8d370 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.341+0000 7f8d9affd640 1 -- 172.21.15.124:0/108897555 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c02e070 con 0x7f8d9c146aa0 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.341+0000 7f8d9affd640 1 -- 172.21.15.124:0/108897555 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c0315a0 con 0x7f8d9c146aa0 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.341+0000 7f8d9affd640 1 -- 172.21.15.124:0/108897555 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c031930 con 0x7f8d9c146aa0 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 -- 172.21.15.124:0/108897555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 msgr2=0x7f8d9c0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 --2- 172.21.15.124:0/108897555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c0b2140 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c0aeb20 tx=0x7f8d8c02b840 comp rx=0 tx=0).stop 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 -- 172.21.15.124:0/108897555 shutdown_connections 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 --2- 172.21.15.124:0/108897555 >> v2:172.21.15.203:3300/0 conn(0x7f8d9c0b0cf0 0x7f8d9c0b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.342 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 --2- 172.21.15.124:0/108897555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c0b2140 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 -- 172.21.15.124:0/108897555 >> 172.21.15.124:0/108897555 conn(0x7f8d9c06a4d0 msgr2=0x7f8d9c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:47.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 -- 172.21.15.124:0/108897555 shutdown_connections 2023-12-13T23:57:47.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.342+0000 7f8da27b0640 1 -- 172.21.15.124:0/108897555 wait complete. 2023-12-13T23:57:47.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.343+0000 7f8da27b0640 1 Processor -- start 2023-12-13T23:57:47.344 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.343+0000 7f8da27b0640 1 -- start start 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0ac8e0 0x7f8d9c1419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0b0cf0 0x7f8d9c157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c159570 con 0x7f8d9c146aa0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c158960 con 0x7f8d9c0b0cf0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da27b0640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d9c158c40 con 0x7f8d9c0ac8e0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41744/0 (socket says 172.21.15.124:41744) 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 -- 172.21.15.124:0/2955841668 learned_addr learned my addr 172.21.15.124:0/2955841668 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8d9b7fe640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0b0cf0 0x7f8d9c157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8d9bfff640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0ac8e0 0x7f8d9c1419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 -- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0ac8e0 msgr2=0x7f8d9c1419e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0ac8e0 0x7f8d9c1419e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 -- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0b0cf0 msgr2=0x7f8d9c157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0b0cf0 0x7f8d9c157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 -- 172.21.15.124:0/2955841668 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c009400 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.344+0000 7f8da0d4d640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400cbc0 tx=0x7f8d9400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400ad40 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d94012400 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d9c158f20 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400aea0 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d9c15d830 con 0x7f8d9c146aa0 2023-12-13T23:57:47.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.345+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d9c15e860 con 0x7f8d9c146aa0 2023-12-13T23:57:47.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.346+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f8d94007070 con 0x7f8d9c146aa0 2023-12-13T23:57:47.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.346+0000 7f8d98ff9640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 0x7f8d8806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:47.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.346+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8d940898d0 con 0x7f8d9c146aa0 2023-12-13T23:57:47.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.347+0000 7f8d9bfff640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 0x7f8d8806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:47.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.347+0000 7f8d9bfff640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 0x7f8d8806d680 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c036000 tx=0x7f8d8c03d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:47.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.351+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f8d940546e0 con 0x7f8d9c146aa0 2023-12-13T23:57:47.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:47.518+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8d9c069400 con 0x7f8d9c146aa0 2023-12-13T23:57:48.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.225+0000 7f8d98ff9640 1 -- 172.21.15.124:0/2955841668 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v52) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8d94017020 con 0x7f8d9c146aa0 2023-12-13T23:57:48.225 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_metadata' removed 2023-12-13T23:57:48.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 msgr2=0x7f8d8806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 0x7f8d8806d680 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c036000 tx=0x7f8d8c03d040 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 msgr2=0x7f8d9c15b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400cbc0 tx=0x7f8d9400a430 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 shutdown_connections 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f8d8806b1c0 0x7f8d8806d680 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f8d9c0b0cf0 0x7f8d9c157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8d9c146aa0 0x7f8d9c15b470 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 --2- 172.21.15.124:0/2955841668 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f8d9c0ac8e0 0x7f8d9c1419e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 >> 172.21.15.124:0/2955841668 conn(0x7f8d9c06a4d0 msgr2=0x7f8d9c1602d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:48.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 shutdown_connections 2023-12-13T23:57:48.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.228+0000 7f8da27b0640 1 -- 172.21.15.124:0/2955841668 wait complete. 2023-12-13T23:57:48.240 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-12-13T23:57:48.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.345+0000 7f78dfc17640 1 Processor -- start 2023-12-13T23:57:48.345 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.345+0000 7f78dfc17640 1 -- start start 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dfc17640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f78d815d330 0x7f78d80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dfc17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d80d2910 0x7f78d80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dfc17640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d80b2b10 con 0x7f78d80d3730 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dfc17640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80d3b50 con 0x7f78d815d330 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dfc17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80d1590 con 0x7f78d80d2910 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dd9b3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f78d815d330 0x7f78d80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78de1b4640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f78d80d3730 0x7f78d80b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34650/0 (socket says 172.21.15.124:34650) 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78de1b4640 1 -- 172.21.15.124:0/2057756808 learned_addr learned my addr 172.21.15.124:0/2057756808 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dd9b3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f78d815d330 0x7f78d80b8b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41756/0 (socket says 172.21.15.124:41756) 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.346+0000 7f78dd1b2640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d80d2910 0x7f78d80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.347 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dd9b3640 1 -- 172.21.15.124:0/2057756808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d80d2910 msgr2=0x7f78d80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dd9b3640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d80d2910 0x7f78d80d31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dd9b3640 1 -- 172.21.15.124:0/2057756808 >> v1:172.21.15.124:6789/0 conn(0x7f78d80d3730 legacy=0x7f78d80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dd9b3640 1 -- 172.21.15.124:0/2057756808 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80d1870 con 0x7f78d815d330 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dc9b1640 1 -- 172.21.15.124:0/2057756808 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750073908 0 0) 0x7f78d80b2b10 con 0x7f78d80d3730 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dd9b3640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d815d330 0x7f78d80b8b80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9049b713ee008754 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.347+0000 7f78dc9b1640 1 -- 172.21.15.124:0/2057756808 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d402e070 con 0x7f78d815d330 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dc9b1640 1 -- 172.21.15.124:0/2057756808 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d40315a0 con 0x7f78d815d330 2023-12-13T23:57:48.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dc9b1640 1 -- 172.21.15.124:0/2057756808 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4031930 con 0x7f78d815d330 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 -- 172.21.15.124:0/2057756808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d815d330 msgr2=0x7f78d80b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d815d330 0x7f78d80b8b80 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d402b840 comp rx=0 tx=0).stop 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 -- 172.21.15.124:0/2057756808 shutdown_connections 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d80d2910 0x7f78d80d31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 --2- 172.21.15.124:0/2057756808 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d815d330 0x7f78d80b8b80 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 -- 172.21.15.124:0/2057756808 >> 172.21.15.124:0/2057756808 conn(0x7f78d8072020 msgr2=0x7f78d8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 -- 172.21.15.124:0/2057756808 shutdown_connections 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.348+0000 7f78dfc17640 1 -- 172.21.15.124:0/2057756808 wait complete. 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 Processor -- start 2023-12-13T23:57:48.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 -- start start 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78d80d3730 0x7f78d80b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d815d330 0x7f78d80b3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d815f050 con 0x7f78d80d2910 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d814bbc0 con 0x7f78d80d3730 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dfc17640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d815e440 con 0x7f78d815d330 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dd9b3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dd9b3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41758/0 (socket says 172.21.15.124:41758) 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.350+0000 7f78dd9b3640 1 -- 172.21.15.124:0/851804731 learned_addr learned my addr 172.21.15.124:0/851804731 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78de1b4640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d815d330 0x7f78d80b3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd1b2640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78d80d3730 0x7f78d80b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 -- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d815d330 msgr2=0x7f78d80b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d815d330 0x7f78d80b3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 -- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78d80d3730 msgr2=0x7f78d80b2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78d80d3730 0x7f78d80b2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 -- 172.21.15.124:0/851804731 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d4009400 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dd9b3640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4004230 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4031f00 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d815e720 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4032640 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.351+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d815ece0 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.352+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d80ce6d0 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.352+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f78d40327a0 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.353+0000 7f78ce7fc640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 0x7f78bc06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.353+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5357+0+0 (secure 0 0 0) 0x7f78d40ab960 con 0x7f78d80d2910 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.353+0000 7f78dd1b2640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 0x7f78bc06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:48.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.354+0000 7f78dd1b2640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 0x7f78bc06d6b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006fd0 tx=0x7f78c8008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:48.356 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.357+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f78d4076770 con 0x7f78d80d2910 2023-12-13T23:57:48.521 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:48.522+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f78d815eea0 con 0x7f78d80d2910 2023-12-13T23:57:49.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.230+0000 7f78ce7fc640 1 -- 172.21.15.124:0/851804731 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v53) v1 ==== 169+0+0 (secure 0 0 0) 0x7f78d4079d90 con 0x7f78d80d2910 2023-12-13T23:57:49.230 INFO:teuthology.orchestra.run.smithi124.stderr:pool 'cephfs_data' removed 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 msgr2=0x7f78bc06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 0x7f78bc06d6b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f78c8006fd0 tx=0x7f78c8008040 comp rx=0 tx=0).stop 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 msgr2=0x7f78d814b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f78d40097a0 tx=0x7f78d40046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 shutdown_connections 2023-12-13T23:57:49.233 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f78bc06b1f0 0x7f78bc06d6b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f78d80d3730 0x7f78d80b2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f78d815d330 0x7f78d80b3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 --2- 172.21.15.124:0/851804731 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f78d80d2910 0x7f78d814b680 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 >> 172.21.15.124:0/851804731 conn(0x7f78d8072020 msgr2=0x7f78d8072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 shutdown_connections 2023-12-13T23:57:49.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.233+0000 7f78dfc17640 1 -- 172.21.15.124:0/851804731 wait complete. 2023-12-13T23:57:49.245 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-12-13T23:57:49.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.350+0000 7fbde4af3640 1 Processor -- start 2023-12-13T23:57:49.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.350+0000 7fbde4af3640 1 -- start start 2023-12-13T23:57:49.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbde4af3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbde4af3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015e730 0x7fbde00b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbde4af3640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde015e2a0 con 0x7fbde015a4e0 2023-12-13T23:57:49.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbde4af3640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde015b520 con 0x7fbde015e730 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbde4af3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde0143870 con 0x7fbde0143490 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015e730 0x7fbde00b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdded9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fbde015a4e0 0x7fbde015a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:34658/0 (socket says 172.21.15.124:34658) 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde015e730 0x7fbde00b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41772/0 (socket says 172.21.15.124:41772) 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdded9d640 1 -- 172.21.15.124:0/325732511 learned_addr learned my addr 172.21.15.124:0/325732511 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 -- 172.21.15.124:0/325732511 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 msgr2=0x7fbde00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 --2- 172.21.15.124:0/325732511 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 -- 172.21.15.124:0/325732511 >> v1:172.21.15.124:6789/0 conn(0x7fbde015a4e0 legacy=0x7fbde015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbddd59a640 1 -- 172.21.15.124:0/325732511 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406674082 0 0) 0x7fbde015e2a0 con 0x7fbde015a4e0 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.351+0000 7fbdddd9b640 1 -- 172.21.15.124:0/325732511 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde015b240 con 0x7fbde015e730 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.352+0000 7fbdddd9b640 1 --2- 172.21.15.124:0/325732511 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00b1230 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40097a0 tx=0x7fbdd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3189718ba6d5eaef server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.352+0000 7fbddd59a640 1 -- 172.21.15.124:0/325732511 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4031830 con 0x7fbde015e730 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.352+0000 7fbddd59a640 1 -- 172.21.15.124:0/325732511 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd4031990 con 0x7fbde015e730 2023-12-13T23:57:49.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.352+0000 7fbddd59a640 1 -- 172.21.15.124:0/325732511 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4031d20 con 0x7fbde015e730 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 -- 172.21.15.124:0/325732511 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 msgr2=0x7fbde00b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 --2- 172.21.15.124:0/325732511 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00b1230 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40097a0 tx=0x7fbdd402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 -- 172.21.15.124:0/325732511 shutdown_connections 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 --2- 172.21.15.124:0/325732511 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00b1230 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 --2- 172.21.15.124:0/325732511 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde00b0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 -- 172.21.15.124:0/325732511 >> 172.21.15.124:0/325732511 conn(0x7fbde006a4d0 msgr2=0x7fbde006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 -- 172.21.15.124:0/325732511 shutdown_connections 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.353+0000 7fbde4af3640 1 -- 172.21.15.124:0/325732511 wait complete. 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.354+0000 7fbde4af3640 1 Processor -- start 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.354+0000 7fbde4af3640 1 -- start start 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 0x7fbde00a8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde01f2110 con 0x7fbde015e730 2023-12-13T23:57:49.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde01f22f0 con 0x7fbde015a4e0 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbde4af3640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde01f25d0 con 0x7fbde0143490 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41786/0 (socket says 172.21.15.124:41786) 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 -- 172.21.15.124:0/1362180991 learned_addr learned my addr 172.21.15.124:0/1362180991 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdde59c640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdddd9b640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 0x7fbde00a8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 -- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 msgr2=0x7fbde013ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 -- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 msgr2=0x7fbde00a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 0x7fbde00a8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdded9d640 1 -- 172.21.15.124:0/1362180991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd4009400 con 0x7fbde015e730 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.355+0000 7fbdddd9b640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 0x7fbde00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbdded9d640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c450 tx=0x7fbdd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd80074e0 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8007660 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde01f28b0 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8004a20 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.356+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde01f2e70 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.357+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde013b090 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.357+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fbdd8004b80 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.357+0000 7fbdceffd640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 0x7fbdc006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.358+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fbdd80888f0 con 0x7fbde015e730 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.358+0000 7fbdde59c640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 0x7fbdc006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.358+0000 7fbdde59c640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 0x7fbdc006d6b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009f20 tx=0x7fbdd00094f0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.362+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fbdd8053960 con 0x7fbde015e730 2023-12-13T23:57:49.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.527+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fbde00677a0 con 0x7fbde015e730 2023-12-13T23:57:49.529 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.530+0000 7fbdceffd640 1 -- 172.21.15.124:0/1362180991 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbdd8056f80 con 0x7fbde015e730 2023-12-13T23:57:49.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 msgr2=0x7fbdc006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 0x7fbdc006d6b0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fbdd0009f20 tx=0x7fbdd00094f0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.532 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 msgr2=0x7fbde00a9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fbdd800c450 tx=0x7fbdd800c910 comp rx=0 tx=0).stop 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 shutdown_connections 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fbdc006b1f0 0x7fbdc006d6b0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fbde015a4e0 0x7fbde00a8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde015e730 0x7fbde00a9100 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 --2- 172.21.15.124:0/1362180991 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fbde0143490 0x7fbde013ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.532+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 >> 172.21.15.124:0/1362180991 conn(0x7fbde006a4d0 msgr2=0x7fbde015a950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.533+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 shutdown_connections 2023-12-13T23:57:49.533 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.533+0000 7fbde4af3640 1 -- 172.21.15.124:0/1362180991 wait complete. 2023-12-13T23:57:49.543 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2023-12-13T23:57:49.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.644+0000 7fe649f7c640 1 Processor -- start 2023-12-13T23:57:49.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.644+0000 7fe649f7c640 1 -- start start 2023-12-13T23:57:49.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.645+0000 7fe649f7c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 0x7fe64415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.645+0000 7fe649f7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.645+0000 7fe649f7c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe64415e2a0 con 0x7fe644143490 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.645+0000 7fe649f7c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe64415b520 con 0x7fe6440b1400 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.645+0000 7fe649f7c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe644143870 con 0x7fe64415e730 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe642ffd640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 0x7fe64415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:41774/0 (socket says 172.21.15.124:41774) 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 -- 172.21.15.124:0/4066373326 learned_addr learned my addr 172.21.15.124:0/4066373326 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe6437fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe644143490 0x7fe6440b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:42126/0 (socket says 172.21.15.124:42126) 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 -- 172.21.15.124:0/4066373326 >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 msgr2=0x7fe64415a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 --2- 172.21.15.124:0/4066373326 >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 0x7fe64415a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 -- 172.21.15.124:0/4066373326 >> v1:172.21.15.203:6789/0 conn(0x7fe644143490 legacy=0x7fe6440b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe643fff640 1 -- 172.21.15.124:0/4066373326 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64415b240 con 0x7fe64415e730 2023-12-13T23:57:49.650 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.646+0000 7fe642ffd640 1 --2- 172.21.15.124:0/4066373326 >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 0x7fe64415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe643fff640 1 --2- 172.21.15.124:0/4066373326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009e50 tx=0x7fe63c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ec9a6721679a66 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe6427fc640 1 -- 172.21.15.124:0/4066373326 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c02cb00 con 0x7fe64415e730 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe6427fc640 1 -- 172.21.15.124:0/4066373326 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c02cc60 con 0x7fe64415e730 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe6427fc640 1 -- 172.21.15.124:0/4066373326 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c0343f0 con 0x7fe64415e730 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 -- 172.21.15.124:0/4066373326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 msgr2=0x7fe64415aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 --2- 172.21.15.124:0/4066373326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009e50 tx=0x7fe63c02bf60 comp rx=0 tx=0).stop 2023-12-13T23:57:49.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 -- 172.21.15.124:0/4066373326 shutdown_connections 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 --2- 172.21.15.124:0/4066373326 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe64415e730 0x7fe64415aa20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 --2- 172.21.15.124:0/4066373326 >> v2:172.21.15.203:3300/0 conn(0x7fe6440b1400 0x7fe64415a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 -- 172.21.15.124:0/4066373326 >> 172.21.15.124:0/4066373326 conn(0x7fe64406a4d0 msgr2=0x7fe64406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 -- 172.21.15.124:0/4066373326 shutdown_connections 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.647+0000 7fe649f7c640 1 -- 172.21.15.124:0/4066373326 wait complete. 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 Processor -- start 2023-12-13T23:57:49.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 -- start start 2023-12-13T23:57:49.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6440b1400 0x7fe6441e74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe644143490 0x7fe6441598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441e6400 con 0x7fe64415e730 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441e66e0 con 0x7fe6440b1400 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.649+0000 7fe649f7c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6441e69c0 con 0x7fe644143490 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41796/0 (socket says 172.21.15.124:41796) 2023-12-13T23:57:49.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 -- 172.21.15.124:0/3963954907 learned_addr learned my addr 172.21.15.124:0/3963954907 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe6437fe640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6440b1400 0x7fe6441e74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe642ffd640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe644143490 0x7fe6441598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 -- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe644143490 msgr2=0x7fe6441598f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe644143490 0x7fe6441598f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 -- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6440b1400 msgr2=0x7fe6441e74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6440b1400 0x7fe6441e74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:49.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 -- 172.21.15.124:0/3963954907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c009400 con 0x7fe64415e730 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe643fff640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009e50 tx=0x7fe63c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c009510 con 0x7fe64415e730 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63c03e450 con 0x7fe64415e730 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63c009b30 con 0x7fe64415e730 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63c03e800 con 0x7fe64415e730 2023-12-13T23:57:49.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.650+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6441e6f80 con 0x7fe64415e730 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.651+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6441e7170 con 0x7fe64415e730 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.652+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe63c03e960 con 0x7fe64415e730 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.652+0000 7fe63bfff640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 0x7fe62406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.652+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe63c0ac3b0 con 0x7fe64415e730 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.652+0000 7fe6437fe640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 0x7fe62406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.653+0000 7fe6437fe640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 0x7fe62406d680 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe634000f60 tx=0x7fe634007c80 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:49.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.656+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe63c077470 con 0x7fe64415e730 2023-12-13T23:57:49.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:49.890+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) v1 -- 0x7fe644069400 con 0x7fe64415e730 2023-12-13T23:57:50.231 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.231+0000 7fe63bfff640 1 -- 172.21.15.124:0/3963954907 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v159) v1 ==== 134+0+0 (secure 0 0 0) 0x7fe63c07aa90 con 0x7fe64415e730 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 msgr2=0x7fe62406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 0x7fe62406d680 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe634000f60 tx=0x7fe634007c80 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 msgr2=0x7fe6441e5890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fe63c009e50 tx=0x7fe63c004a40 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 shutdown_connections 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe62406b1c0 0x7fe62406d680 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe64415e730 0x7fe6441e5890 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe644143490 0x7fe6441598f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 --2- 172.21.15.124:0/3963954907 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6440b1400 0x7fe6441e74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:50.234 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 >> 172.21.15.124:0/3963954907 conn(0x7fe64406a4d0 msgr2=0x7fe64406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:50.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 shutdown_connections 2023-12-13T23:57:50.235 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:50.234+0000 7fe649f7c640 1 -- 172.21.15.124:0/3963954907 wait complete. 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner: 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2023-12-13T23:57:50.246 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs/mount.py", line 775, in run_python 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: p.wait() 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:Ran 2 tests in 127.882s 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner: 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:====================================================================== 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:57:50.247 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs/mount.py", line 775, in run_python 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: p.wait() 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi124 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2023-12-13T23:57:50.248 INFO:tasks.cephfs_test_runner: 2023-12-13T23:57:50.248 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:57:50.338 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=18d2041c5ebc45a3905a4b3df3c8dfa5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_021ac1670a21c19759080c6da90baaeb42e7d175/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2023-12-13T23:57:50.340 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-12-13T23:57:50.351 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-12-13T23:57:50.387 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-12-13T23:57:50.387 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:50.388 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:50.407 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:50.408 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-12-13T23:57:50.408 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:50.464 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056363,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=154 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248848k,nr_inodes=812212,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497700k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-12-13T23:57:50.464 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:50.464 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:50.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:50.483 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2023-12-13T23:57:50.483 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/proc/self/mounts of=/dev/stdout 2023-12-13T23:57:50.541 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056360,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=394 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497696k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-12-13T23:57:50.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.542 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-12-13T23:57:50.557 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-12-13T23:57:50.557 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-12-13T23:57:50.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.558 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-12-13T23:57:50.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.625 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-12-13T23:57:50.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.694 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-12-13T23:57:50.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.797 DEBUG:teuthology.orchestra.run.smithi203:> ip netns list 2023-12-13T23:57:50.816 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-12-13T23:57:50.816 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-12-13T23:57:50.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-12-13T23:57:50.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.887 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-12-13T23:57:50.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-12-13T23:57:50.958 DEBUG:teuthology.orchestra.run.smithi203:> sudo ip link delete ceph-brx 2023-12-13T23:57:51.055 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-13T23:57:51.064 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-13T23:57:51.064 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:51.167 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.168+0000 7fe707125640 1 Processor -- start 2023-12-13T23:57:51.168 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.168+0000 7fe707125640 1 -- start start 2023-12-13T23:57:51.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe707125640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe7000b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe707125640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe700071770 con 0x7fe7000ceb70 2023-12-13T23:57:51.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe707125640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7000ccb60 con 0x7fe700144980 2023-12-13T23:57:51.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe707125640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70015ced0 con 0x7fe7000b4370 2023-12-13T23:57:51.171 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f7fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fe7000ceb70 0x7fe7000cc450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41012/0 (socket says 172.21.15.124:41012) 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f7fff640 1 -- 172.21.15.124:0/3281321984 learned_addr learned my addr 172.21.15.124:0/3281321984 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406679046 0 0) 0x7fe700071770 con 0x7fe7000ceb70 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6e8003520 con 0x7fe7000ceb70 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe7056c2640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe7000b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3599939458 0 0) 0x7fe6e8003520 con 0x7fe7000ceb70 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 msgr2=0x7fe7000b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe7000b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 >> v1:172.21.15.203:6789/0 conn(0x7fe700144980 legacy=0x7fe70015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.172 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7000b4ce0 con 0x7fe7000ceb70 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fe6f0001670 con 0x7fe7000ceb70 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 >> v1:172.21.15.124:6789/0 conn(0x7fe7000ceb70 legacy=0x7fe7000cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe6e80039a0 0x7fe6e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe6e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8004d00 0x7fe6e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70015ced0 con 0x7fe6e80039a0 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e800ba30 con 0x7fe6e8004d00 2023-12-13T23:57:51.173 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.169+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e800bbf0 con 0x7fe7000b4370 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe6e80039a0 0x7fe6e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe7056c2640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8004d00 0x7fe6e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe6f7fff640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe6e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 -- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 msgr2=0x7fe6e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000b4370 0x7fe6e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 -- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8004d00 msgr2=0x7fe6e8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e8004d00 0x7fe6e8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.170+0000 7fe704ec1640 1 -- 172.21.15.124:0/3281321984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8003520 con 0x7fe6e80039a0 2023-12-13T23:57:51.174 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe704ec1640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe6e80039a0 0x7fe6e8003de0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc001720 tx=0x7fe6e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ebca0ee68e1f837 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0052d0 con 0x7fe6e80039a0 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc02e050 con 0x7fe6e80039a0 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe6f77fe640 1 -- 172.21.15.124:0/3281321984 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc0035f0 con 0x7fe6e80039a0 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 -- 172.21.15.124:0/3281321984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe6e80039a0 msgr2=0x7fe6e8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 --2- 172.21.15.124:0/3281321984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe6e80039a0 0x7fe6e8003de0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc001720 tx=0x7fe6e80057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe7056c2640 1 -- 172.21.15.124:0/3281321984 reap_dead start 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 -- 172.21.15.124:0/3281321984 shutdown_connections 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 -- 172.21.15.124:0/3281321984 >> 172.21.15.124:0/3281321984 conn(0x7fe70006a4d0 msgr2=0x7fe70006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.175 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 -- 172.21.15.124:0/3281321984 shutdown_connections 2023-12-13T23:57:51.176 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.171+0000 7fe707125640 1 -- 172.21.15.124:0/3281321984 wait complete. 2023-12-13T23:57:51.178 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.172+0000 7fe707125640 1 Processor -- start 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.172+0000 7fe707125640 1 -- start start 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 0x7fe7000a8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000c9390 0x7fe7000c9810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe700141ee0 con 0x7fe700141520 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7000ca390 con 0x7fe6e80039a0 2023-12-13T23:57:51.179 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe707125640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7000ca0b0 con 0x7fe7000c9390 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe6f7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe6f7fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41326/0 (socket says 172.21.15.124:41326) 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe6f7fff640 1 -- 172.21.15.124:0/1410313524 learned_addr learned my addr 172.21.15.124:0/1410313524 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.173+0000 7fe704ec1640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 0x7fe7000a8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe7056c2640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000c9390 0x7fe7000c9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.180 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 -- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000c9390 msgr2=0x7fe7000c9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000c9390 0x7fe7000c9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 -- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 msgr2=0x7fe7000a8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.181 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 0x7fe7000a8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 -- 172.21.15.124:0/1410313524 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7000b2200 con 0x7fe700141520 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe704ec1640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 0x7fe7000a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f7fff640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00072c0 tx=0x7fe6f0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000f8e0 con 0x7fe700141520 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f000fa40 con 0x7fe700141520 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7000b24e0 con 0x7fe700141520 2023-12-13T23:57:51.182 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f000fdf0 con 0x7fe700141520 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.174+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7001f2ec0 con 0x7fe700141520 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.175+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe700069400 con 0x7fe700141520 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.175+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe6f001d460 con 0x7fe700141520 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.176+0000 7fe6f57fa640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 0x7fe6ec06d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.176+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe6f0089870 con 0x7fe700141520 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.176+0000 7fe704ec1640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 0x7fe6ec06d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.183 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.176+0000 7fe704ec1640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 0x7fe6ec06d6b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc033000 tx=0x7fe6fc009030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.184 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.180+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe6f00548a0 con 0x7fe700141520 2023-12-13T23:57:51.343 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.344+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7000cd380 con 0x7fe6ec06b1f0 2023-12-13T23:57:51.346 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.347+0000 7fe6f57fa640 1 -- 172.21.15.124:0/1410313524 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51153 (secure 0 0 0) 0x7fe6fc005070 con 0x7fe6ec06b1f0 2023-12-13T23:57:51.346 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:51.347 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:51.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 msgr2=0x7fe6ec06d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 0x7fe6ec06d6b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc033000 tx=0x7fe6fc009030 comp rx=0 tx=0).stop 2023-12-13T23:57:51.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 msgr2=0x7fe7001419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fe6f00072c0 tx=0x7fe6f0007780 comp rx=0 tx=0).stop 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 shutdown_connections 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe6ec06b1f0 0x7fe6ec06d6b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe7000c9390 0x7fe7000c9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe700141520 0x7fe7001419a0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 --2- 172.21.15.124:0/1410313524 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe6e80039a0 0x7fe7000a8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 >> 172.21.15.124:0/1410313524 conn(0x7fe70006a4d0 msgr2=0x7fe700071b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 shutdown_connections 2023-12-13T23:57:51.350 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.349+0000 7fe707125640 1 -- 172.21.15.124:0/1410313524 wait complete. 2023-12-13T23:57:51.359 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2023-12-13T23:57:50.148507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":226416,"kb_used_data":6072,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748304,"statfs":{"total":773094113280,"available":772862263296,"internally_reserved":0,"allocated":6217728,"data_stored":3266727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12591,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001600"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-13T23:57:48.229026+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:48.229026+0000","last_peered":"2023-12-13T23:57:48.229026+0000","last_clean":"2023-12-13T23:57:48.229026+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:48.229026+0000","last_undegraded":"2023-12-13T23:57:48.229026+0000","last_fullsized":"2023-12-13T23:57:48.229026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60999999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.29099999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.47499999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.443}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.58099999999999996}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.625}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63700000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.498},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.51500000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.377},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.627}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.37}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.432}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48899999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.498}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.626},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.375},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59299999999999997}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.628},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.5}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.38}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.33000000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.749},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.628}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.70899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":849030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.36199999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.32400000000000001}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.373},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.65100000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:51.361 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:51.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.461+0000 7fdbc087c640 1 Processor -- start 2023-12-13T23:57:51.461 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.461+0000 7fdbc087c640 1 -- start start 2023-12-13T23:57:51.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbc087c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdbbc0d2910 0x7fdbbc0d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.462 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbc087c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d3730 0x7fdbbc0b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbc087c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc0b2b10 con 0x7fdbbc15d330 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbc087c640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d1590 con 0x7fdbbc0d2910 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbc087c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0d1870 con 0x7fdbbc0d3730 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbba59c640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fdbbc15d330 0x7fdbbc0b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41014/0 (socket says 172.21.15.124:41014) 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbba59c640 1 -- 172.21.15.124:0/4137801325 learned_addr learned my addr 172.21.15.124:0/4137801325 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.462+0000 7fdbb9d9b640 1 --2- 172.21.15.124:0/4137801325 >> v2:172.21.15.124:3300/0 conn(0x7fdbbc0d2910 0x7fdbbc0d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbbad9d640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d3730 0x7fdbbc0b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbb9d9b640 1 -- 172.21.15.124:0/4137801325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d3730 msgr2=0x7fdbbc0b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:51.463 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbb9d9b640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d3730 0x7fdbbc0b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbb9d9b640 1 -- 172.21.15.124:0/4137801325 >> v1:172.21.15.124:6789/0 conn(0x7fdbbc15d330 legacy=0x7fdbbc0b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbb959a640 1 -- 172.21.15.124:0/4137801325 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378393294 0 0) 0x7fdbbc0b2b10 con 0x7fdbbc15d330 2023-12-13T23:57:51.464 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.463+0000 7fdbb9d9b640 1 -- 172.21.15.124:0/4137801325 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0d3b90 con 0x7fdbbc0d2910 2023-12-13T23:57:51.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbb9d9b640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc0d2910 0x7fdbbc0d31f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009d90 tx=0x7fdbb002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85619599fdc560e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbb959a640 1 -- 172.21.15.124:0/4137801325 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb002cbc0 con 0x7fdbbc0d2910 2023-12-13T23:57:51.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbb959a640 1 -- 172.21.15.124:0/4137801325 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb002cd20 con 0x7fdbbc0d2910 2023-12-13T23:57:51.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbb959a640 1 -- 172.21.15.124:0/4137801325 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0034660 con 0x7fdbbc0d2910 2023-12-13T23:57:51.467 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 -- 172.21.15.124:0/4137801325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc0d2910 msgr2=0x7fdbbc0d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc0d2910 0x7fdbbc0d31f0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0009d90 tx=0x7fdbb002be10 comp rx=0 tx=0).stop 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 -- 172.21.15.124:0/4137801325 shutdown_connections 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d3730 0x7fdbbc0b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 --2- 172.21.15.124:0/4137801325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc0d2910 0x7fdbbc0d31f0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 -- 172.21.15.124:0/4137801325 >> 172.21.15.124:0/4137801325 conn(0x7fdbbc072020 msgr2=0x7fdbbc072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 -- 172.21.15.124:0/4137801325 shutdown_connections 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.464+0000 7fdbc087c640 1 -- 172.21.15.124:0/4137801325 wait complete. 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 Processor -- start 2023-12-13T23:57:51.468 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 -- start start 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d2910 0x7fdbbc14b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbbc0d3730 0x7fdbbc15d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc0c70b0 con 0x7fdbbc15d330 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc14bbc0 con 0x7fdbbc0d3730 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbc087c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc157eb0 con 0x7fdbbc0d2910 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.466+0000 7fdbbad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.469 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41346/0 (socket says 172.21.15.124:41346) 2023-12-13T23:57:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 -- 172.21.15.124:0/94493262 learned_addr learned my addr 172.21.15.124:0/94493262 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbba59c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d2910 0x7fdbbc14b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbb9d9b640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbbc0d3730 0x7fdbbc15d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.470 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 -- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d2910 msgr2=0x7fdbbc14b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d2910 0x7fdbbc14b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 -- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbbc0d3730 msgr2=0x7fdbbc15d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbbc0d3730 0x7fdbbc15d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 -- 172.21.15.124:0/94493262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb0009a70 con 0x7fdbbc15d330 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbbad9d640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d800 tx=0x7fdbac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00b6c0 con 0x7fdbbc15d330 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac00b840 con 0x7fdbbc15d330 2023-12-13T23:57:51.471 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc154fc0 con 0x7fdbbc15d330 2023-12-13T23:57:51.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00bbf0 con 0x7fdbbc15d330 2023-12-13T23:57:51.472 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.467+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc157bd0 con 0x7fdbbc15d330 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.468+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc0c71f0 con 0x7fdbbc15d330 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.468+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fdbac00bd50 con 0x7fdbbc15d330 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.469+0000 7fdbaaffd640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 0x7fdb9c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.469+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fdbac089bf0 con 0x7fdbbc15d330 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.469+0000 7fdbba59c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 0x7fdb9c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.473 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.470+0000 7fdbba59c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 0x7fdb9c06d680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f60 tx=0x7fdbb4007c00 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.474 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.473+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fdbbc0c71f0 con 0x7fdbbc15d330 2023-12-13T23:57:51.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.638+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdbbc0d2110 con 0x7fdb9c06b1c0 2023-12-13T23:57:51.640 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.640+0000 7fdbaaffd640 1 -- 172.21.15.124:0/94493262 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51153 (secure 0 0 0) 0x7fdbb4007270 con 0x7fdb9c06b1c0 2023-12-13T23:57:51.641 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:51.641 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:51.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 msgr2=0x7fdb9c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 0x7fdb9c06d680 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000f60 tx=0x7fdbb4007c00 comp rx=0 tx=0).stop 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 msgr2=0x7fdbbc15dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00d800 tx=0x7fdbac00dcc0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 shutdown_connections 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fdb9c06b1c0 0x7fdb9c06d680 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fdbbc0d3730 0x7fdbbc15d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdbbc15d330 0x7fdbbc15dc50 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 --2- 172.21.15.124:0/94493262 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fdbbc0d2910 0x7fdbbc14b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 >> 172.21.15.124:0/94493262 conn(0x7fdbbc072020 msgr2=0x7fdbbc072400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 shutdown_connections 2023-12-13T23:57:51.644 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.642+0000 7fdbc087c640 1 -- 172.21.15.124:0/94493262 wait complete. 2023-12-13T23:57:51.652 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2023-12-13T23:57:50.148507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":226416,"kb_used_data":6072,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748304,"statfs":{"total":773094113280,"available":772862263296,"internally_reserved":0,"allocated":6217728,"data_stored":3266727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12591,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001600"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-13T23:57:48.229026+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:48.229026+0000","last_peered":"2023-12-13T23:57:48.229026+0000","last_clean":"2023-12-13T23:57:48.229026+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:48.229026+0000","last_undegraded":"2023-12-13T23:57:48.229026+0000","last_fullsized":"2023-12-13T23:57:48.229026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60999999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.29099999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.47499999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.443}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.58099999999999996}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.625}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63700000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.498},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.51500000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.377},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.627}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.37}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.432}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48899999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.498}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.626},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.375},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59299999999999997}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.628},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.5}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.38}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.33000000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.749},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.628}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.70899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":849030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.36199999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.32400000000000001}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.373},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.65100000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:51.654 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-13T23:57:51.654 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:51.754 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 Processor -- start 2023-12-13T23:57:51.755 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 -- start start 2023-12-13T23:57:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f24040d2910 0x7f24040d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24040b2b10 con 0x7f240415d330 2023-12-13T23:57:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24040d1590 con 0x7f24040d3730 2023-12-13T23:57:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.755+0000 7f2409bde640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24040d3bc0 con 0x7f24040d2910 2023-12-13T23:57:51.756 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f24037fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f240415d330 0x7f24040b8b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41022/0 (socket says 172.21.15.124:41022) 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f24037fe640 1 -- 172.21.15.124:0/2930720458 learned_addr learned my addr 172.21.15.124:0/2930720458 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f2402ffd640 1 --2- 172.21.15.124:0/2930720458 >> v2:172.21.15.124:3300/0 conn(0x7f24040d2910 0x7f24040d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f2402ffd640 1 -- 172.21.15.124:0/2930720458 >> v1:172.21.15.203:6789/0 conn(0x7f24040d3730 legacy=0x7f24040b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f2402ffd640 1 -- 172.21.15.124:0/2930720458 >> v1:172.21.15.124:6789/0 conn(0x7f240415d330 legacy=0x7f24040b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f24027fc640 1 -- 172.21.15.124:0/2930720458 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1113446313 0 0) 0x7f24040b2b10 con 0x7f240415d330 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.756+0000 7f2402ffd640 1 -- 172.21.15.124:0/2930720458 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24040d1870 con 0x7f24040d2910 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2402ffd640 1 --2- 172.21.15.124:0/2930720458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24040d2910 0x7f24040d31f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f23f80097a0 tx=0x7f23f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc519d46f3163e01 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.757 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f24027fc640 1 -- 172.21.15.124:0/2930720458 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8031830 con 0x7f24040d2910 2023-12-13T23:57:51.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f24027fc640 1 -- 172.21.15.124:0/2930720458 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f8031990 con 0x7f24040d2910 2023-12-13T23:57:51.758 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f24027fc640 1 -- 172.21.15.124:0/2930720458 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8031d20 con 0x7f24040d2910 2023-12-13T23:57:51.760 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 -- 172.21.15.124:0/2930720458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24040d2910 msgr2=0x7f24040d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 --2- 172.21.15.124:0/2930720458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24040d2910 0x7f24040d31f0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f23f80097a0 tx=0x7f23f802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 -- 172.21.15.124:0/2930720458 shutdown_connections 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 --2- 172.21.15.124:0/2930720458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f24040d2910 0x7f24040d31f0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 -- 172.21.15.124:0/2930720458 >> 172.21.15.124:0/2930720458 conn(0x7f2404072020 msgr2=0x7f2404072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 -- 172.21.15.124:0/2930720458 shutdown_connections 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.757+0000 7f2409bde640 1 -- 172.21.15.124:0/2930720458 wait complete. 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.758+0000 7f2409bde640 1 Processor -- start 2023-12-13T23:57:51.761 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 -- start start 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 0x7f24040b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 0x7f24040c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24040b1c70 con 0x7f240415d330 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240415ba50 con 0x7f24040d2910 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2409bde640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240414c160 con 0x7f24040d3730 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2403fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2403fff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41354/0 (socket says 172.21.15.124:41354) 2023-12-13T23:57:51.762 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.759+0000 7f2403fff640 1 -- 172.21.15.124:0/1329306984 learned_addr learned my addr 172.21.15.124:0/1329306984 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f24037fe640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 0x7f24040b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2402ffd640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 0x7f24040c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 -- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 msgr2=0x7f24040c85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 0x7f24040c85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 -- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 msgr2=0x7f24040b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 0x7f24040b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 -- 172.21.15.124:0/1329306984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009400 con 0x7f240415d330 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2402ffd640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 0x7f24040c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:51.763 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f24037fe640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 0x7f24040b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2403fff640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7a0 tx=0x7f23fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc003b70 con 0x7f240415d330 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23fc003cf0 con 0x7f240415d330 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24040c9150 con 0x7f240415d330 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23fc003ea0 con 0x7f240415d330 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.760+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240414c420 con 0x7f240415d330 2023-12-13T23:57:51.764 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.761+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2404069400 con 0x7f240415d330 2023-12-13T23:57:51.765 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.762+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f23fc00aa90 con 0x7f240415d330 2023-12-13T23:57:51.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.762+0000 7f23ebfff640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 0x7f23f006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:51.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.762+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f23fc089580 con 0x7f240415d330 2023-12-13T23:57:51.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.762+0000 7f24037fe640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 0x7f23f006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:51.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.763+0000 7f24037fe640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 0x7f23f006d680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f23f4007fd0 tx=0x7f23f4007dc0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:51.766 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.766+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f23fc0545b0 con 0x7f240415d330 2023-12-13T23:57:51.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.932+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24040d2110 con 0x7f23f006b1c0 2023-12-13T23:57:51.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.934+0000 7f23ebfff640 1 -- 172.21.15.124:0/1329306984 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51153 (secure 0 0 0) 0x7f23f4005d40 con 0x7f23f006b1c0 2023-12-13T23:57:51.934 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:51.934 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:51.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 msgr2=0x7f23f006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 0x7f23f006d680 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f23f4007fd0 tx=0x7f23f4007dc0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 msgr2=0x7f24040c8af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f23fc00c7a0 tx=0x7f23fc00cc60 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 shutdown_connections 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f23f006b1c0 0x7f23f006d680 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f240415d330 0x7f24040c8af0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f24040d3730 0x7f24040c85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 --2- 172.21.15.124:0/1329306984 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f24040d2910 0x7f24040b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 >> 172.21.15.124:0/1329306984 conn(0x7f2404072020 msgr2=0x7f24040b55e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:51.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 shutdown_connections 2023-12-13T23:57:51.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:51.936+0000 7f2409bde640 1 -- 172.21.15.124:0/1329306984 wait complete. 2023-12-13T23:57:51.946 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2023-12-13T23:57:50.148507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":6,"kb":754974720,"kb_used":226416,"kb_used_data":6072,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748304,"statfs":{"total":773094113280,"available":772862263296,"internally_reserved":0,"allocated":6217728,"data_stored":3266727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12591,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-55,"num_write_kb":-31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001600"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-13T23:57:48.229026+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:48.229026+0000","last_peered":"2023-12-13T23:57:48.229026+0000","last_clean":"2023-12-13T23:57:48.229026+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:48.229026+0000","last_undegraded":"2023-12-13T23:57:48.229026+0000","last_fullsized":"2023-12-13T23:57:48.229026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574886,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58699999999999997}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60999999999999999}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.69099999999999995}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.29099999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.47499999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.34300000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.443}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.58099999999999996}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.501},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.625}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.63700000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.65000000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.42599999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.58099999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.44600000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.498},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.51500000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.59999999999999998}]}]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.377},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.43099999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.65000000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.627}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.66500000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574886,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.37}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.432}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.48899999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.61299999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.498}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.626},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.55800000000000005}]}]},{"osd":2,"up_from":13,"seq":55834574887,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.375},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.26400000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.32000000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.59299999999999997}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.628},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.5}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53000000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574887,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":258662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.38}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.33000000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.749},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.628}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.53800000000000003}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.70899999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574886,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28212,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343628,"statfs":{"total":96636764160,"available":96607875072,"internally_reserved":0,"allocated":1216512,"data_stored":849030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.36199999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.32400000000000001}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.373},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.77900000000000003}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.65100000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.71399999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:51.948 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-13T23:57:52.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.047+0000 7ff3149ce640 1 Processor -- start 2023-12-13T23:57:52.047 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.047+0000 7ff3149ce640 1 -- start start 2023-12-13T23:57:52.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff3149ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:52.048 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff3149ce640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100b2b10 con 0x7ff3100d2d90 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff3149ce640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3100d1590 con 0x7ff3100d2910 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff3149ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100d1870 con 0x7ff31015d330 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff30ed9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ff3100d2d90 0x7ff3100b6390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41028/0 (socket says 172.21.15.124:41028) 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.048+0000 7ff30ed9d640 1 -- 172.21.15.124:0/3778542545 learned_addr learned my addr 172.21.15.124:0/3778542545 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30e59c640 1 --2- 172.21.15.124:0/3778542545 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704987495 0 0) 0x7ff3100b2b10 con 0x7ff3100d2d90 2023-12-13T23:57:52.049 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f8003520 con 0x7ff3100d2d90 2023-12-13T23:57:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30e59c640 1 -- 172.21.15.124:0/3778542545 >> v1:172.21.15.203:6789/0 conn(0x7ff3100d2910 legacy=0x7ff3100d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713045986 0 0) 0x7ff3100d1590 con 0x7ff3100d2910 2023-12-13T23:57:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30e59c640 1 -- 172.21.15.124:0/3778542545 >> v1:172.21.15.124:6789/0 conn(0x7ff3100d2d90 legacy=0x7ff3100b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.050 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.049+0000 7ff30e59c640 1 -- 172.21.15.124:0/3778542545 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3100d1b50 con 0x7ff31015d330 2023-12-13T23:57:52.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff30e59c640 1 --2- 172.21.15.124:0/3778542545 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100b8b80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff300009dc0 tx=0x7ff30002be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3651d5d396853a server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:52.051 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30002cc40 con 0x7ff31015d330 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30002cda0 con 0x7ff31015d330 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff30d59a640 1 -- 172.21.15.124:0/3778542545 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300034640 con 0x7ff31015d330 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff3149ce640 1 -- 172.21.15.124:0/3778542545 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 msgr2=0x7ff3100b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.050+0000 7ff3149ce640 1 --2- 172.21.15.124:0/3778542545 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100b8b80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff300009dc0 tx=0x7ff30002be90 comp rx=0 tx=0).stop 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.051+0000 7ff3149ce640 1 -- 172.21.15.124:0/3778542545 shutdown_connections 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.051+0000 7ff3149ce640 1 --2- 172.21.15.124:0/3778542545 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100b8b80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.051+0000 7ff3149ce640 1 -- 172.21.15.124:0/3778542545 >> 172.21.15.124:0/3778542545 conn(0x7ff310072020 msgr2=0x7ff310072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.051+0000 7ff3149ce640 1 -- 172.21.15.124:0/3778542545 shutdown_connections 2023-12-13T23:57:52.052 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.051+0000 7ff3149ce640 1 -- 172.21.15.124:0/3778542545 wait complete. 2023-12-13T23:57:52.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.052+0000 7ff3149ce640 1 Processor -- start 2023-12-13T23:57:52.053 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.052+0000 7ff3149ce640 1 -- start start 2023-12-13T23:57:52.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:52.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3100d2d90 0x7ff3100cad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:52.055 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100c9320 con 0x7ff3100d2910 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100c86a0 con 0x7ff3100d2d90 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3100c8980 con 0x7ff31015d330 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41356/0 (socket says 172.21.15.124:41356) 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 -- 172.21.15.124:0/608935077 learned_addr learned my addr 172.21.15.124:0/608935077 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30dd9b640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3100d2d90 0x7ff3100cad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30ed9d640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:52.056 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 -- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 msgr2=0x7ff3100cb260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100cb260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 -- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3100d2d90 msgr2=0x7ff3100cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3100d2d90 0x7ff3100cad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 -- 172.21.15.124:0/608935077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff300009a70 con 0x7ff3100d2910 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff30e59c640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff300000f80 tx=0x7ff300004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30003e610 con 0x7ff3100d2910 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30003e770 con 0x7ff3100d2910 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.053+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3100c8c60 con 0x7ff3100d2910 2023-12-13T23:57:52.057 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.054+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff30003eae0 con 0x7ff3100d2910 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.054+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3100c8360 con 0x7ff3100d2910 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.054+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3100d3b10 con 0x7ff3100d2910 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.055+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff30003b050 con 0x7ff3100d2910 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.055+0000 7ff2feffd640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 0x7ff2f006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.055+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff3000abc10 con 0x7ff3100d2910 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.055+0000 7ff30dd9b640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 0x7ff2f006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:52.058 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.056+0000 7ff30dd9b640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 0x7ff2f006d680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3100c6b30 tx=0x7ff304009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:52.059 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.059+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7ff300076e10 con 0x7ff3100d2910 2023-12-13T23:57:52.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.226+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff310069400 con 0x7ff3100d2910 2023-12-13T23:57:52.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.227+0000 7ff2feffd640 1 -- 172.21.15.124:0/608935077 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+13049 (secure 0 0 0) 0x7ff300039090 con 0x7ff3100d2910 2023-12-13T23:57:52.227 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:52.227 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":53,"fsid":"f45f9ea9-91cf-4589-b298-b4d7e1fc4f09","created":"2023-12-13T23:54:47.988068+0000","modified":"2023-12-13T23:57:49.226874+0000","last_up_change":"2023-12-13T23:54:59.130922+0000","last_in_change":"2023-12-13T23:54:50.241560+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-13T23:55:00.158578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"27b9dad0-42a9-4d72-8917-1fef77b307ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6818","nonce":2614715067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6822","nonce":2614715067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6829","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6831","nonce":2614715067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":2614715067},{"type":"v1","addr":"172.21.15.124:6827","nonce":2614715067}]},"public_addr":"172.21.15.124:6818/2614715067","cluster_addr":"172.21.15.124:6822/2614715067","heartbeat_back_addr":"172.21.15.124:6831/2614715067","heartbeat_front_addr":"172.21.15.124:6827/2614715067","state":["exists","up"]},{"osd":1,"uuid":"bcdb3444-af55-4d8e-9e7b-d8f23ea24f51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6801","nonce":3276151417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6803","nonce":3276151417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6807","nonce":3276151417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":3276151417},{"type":"v1","addr":"172.21.15.124:6805","nonce":3276151417}]},"public_addr":"172.21.15.124:6801/3276151417","cluster_addr":"172.21.15.124:6803/3276151417","heartbeat_back_addr":"172.21.15.124:6807/3276151417","heartbeat_front_addr":"172.21.15.124:6805/3276151417","state":["exists","up"]},{"osd":2,"uuid":"72e1a62d-472d-4db0-98c0-ff0c9e4e6d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6809","nonce":1670235117}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6811","nonce":1670235117}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6815","nonce":1670235117}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":1670235117},{"type":"v1","addr":"172.21.15.124:6813","nonce":1670235117}]},"public_addr":"172.21.15.124:6809/1670235117","cluster_addr":"172.21.15.124:6811/1670235117","heartbeat_back_addr":"172.21.15.124:6815/1670235117","heartbeat_front_addr":"172.21.15.124:6813/1670235117","state":["exists","up"]},{"osd":3,"uuid":"12ee35f6-b6ed-4265-9e69-d9de43ea99fc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6817","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6819","nonce":3893379194}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6821","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6823","nonce":3893379194}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6830","nonce":3893379194}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6825","nonce":3893379194},{"type":"v1","addr":"172.21.15.124:6826","nonce":3893379194}]},"public_addr":"172.21.15.124:6819/3893379194","cluster_addr":"172.21.15.124:6823/3893379194","heartbeat_back_addr":"172.21.15.124:6830/3893379194","heartbeat_front_addr":"172.21.15.124:6826/3893379194","state":["exists","up"]},{"osd":4,"uuid":"1a1f90f7-9d73-4e7c-9797-3cdcc1ced36b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6809","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6811","nonce":3737544324}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6814","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6816","nonce":3737544324}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6822","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6823","nonce":3737544324}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6819","nonce":3737544324},{"type":"v1","addr":"172.21.15.203:6821","nonce":3737544324}]},"public_addr":"172.21.15.203:6811/3737544324","cluster_addr":"172.21.15.203:6816/3737544324","heartbeat_back_addr":"172.21.15.203:6823/3737544324","heartbeat_front_addr":"172.21.15.203:6821/3737544324","state":["exists","up"]},{"osd":5,"uuid":"e7ce641b-9ed7-4361-a39b-5551b768c71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6824","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6825","nonce":1613580500}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6826","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6827","nonce":1613580500}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6830","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6831","nonce":1613580500}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6828","nonce":1613580500},{"type":"v1","addr":"172.21.15.203:6829","nonce":1613580500}]},"public_addr":"172.21.15.203:6825/1613580500","cluster_addr":"172.21.15.203:6827/1613580500","heartbeat_back_addr":"172.21.15.203:6831/1613580500","heartbeat_front_addr":"172.21.15.203:6829/1613580500","state":["exists","up"]},{"osd":6,"uuid":"c952a793-f8b6-4ac8-ad3a-e0a9f65ad2e7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6808","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6810","nonce":3358010181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6812","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6813","nonce":3358010181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6818","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6820","nonce":3358010181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6815","nonce":3358010181},{"type":"v1","addr":"172.21.15.203:6817","nonce":3358010181}]},"public_addr":"172.21.15.203:6810/3358010181","cluster_addr":"172.21.15.203:6813/3358010181","heartbeat_back_addr":"172.21.15.203:6820/3358010181","heartbeat_front_addr":"172.21.15.203:6817/3358010181","state":["exists","up"]},{"osd":7,"uuid":"288daefa-555d-4bea-99c3-57ca8b745384","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6800","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6801","nonce":2404558368}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6802","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6803","nonce":2404558368}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6806","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6807","nonce":2404558368}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6804","nonce":2404558368},{"type":"v1","addr":"172.21.15.203:6805","nonce":2404558368}]},"public_addr":"172.21.15.203:6801/2404558368","cluster_addr":"172.21.15.203:6803/2404558368","heartbeat_back_addr":"172.21.15.203:6807/2404558368","heartbeat_front_addr":"172.21.15.203:6805/2404558368","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.417833+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.381458+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.346391+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.390087+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.375710+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.437067+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.358731+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701272860131327,"old_weight":0,"last_purged_snaps_scrub":"2023-12-13T23:54:55.316332+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.203:6837/2421641365":"2023-12-14T23:57:46.217772+0000","172.21.15.203:6835/2421641365":"2023-12-14T23:57:46.217772+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 msgr2=0x7ff2f006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 0x7ff2f006d680 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff3100c6b30 tx=0x7ff304009040 comp rx=0 tx=0).stop 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 msgr2=0x7ff3101526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7ff300000f80 tx=0x7ff300004a40 comp rx=0 tx=0).stop 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 shutdown_connections 2023-12-13T23:57:52.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff2f006b1c0 0x7ff2f006d680 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff3100d2d90 0x7ff3100cad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff31015d330 0x7ff3100cb260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 --2- 172.21.15.124:0/608935077 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3100d2910 0x7ff3101526f0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 >> 172.21.15.124:0/608935077 conn(0x7ff310072020 msgr2=0x7ff31015c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 shutdown_connections 2023-12-13T23:57:52.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:52.229+0000 7ff3149ce640 1 -- 172.21.15.124:0/608935077 wait complete. 2023-12-13T23:57:53.241 INFO:tasks.ceph:Scrubbing osd.0 2023-12-13T23:57:53.241 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:53.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.355+0000 7ff958448640 1 Processor -- start 2023-12-13T23:57:53.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.355+0000 7ff958448640 1 -- start start 2023-12-13T23:57:53.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff958448640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7ff950143490 0x7ff9500b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff958448640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9500b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff958448640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95015e2a0 con 0x7ff95015a4e0 2023-12-13T23:57:53.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff958448640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95015b520 con 0x7ff950143490 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff958448640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff950143870 con 0x7ff95015e730 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff9569e5640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7ff95015a4e0 0x7ff95015a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41040/0 (socket says 172.21.15.124:41040) 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.356+0000 7ff9569e5640 1 -- 172.21.15.124:0/1737343986 learned_addr learned my addr 172.21.15.124:0/1737343986 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1737343986 >> v2:172.21.15.203:3300/0 conn(0x7ff950143490 0x7ff9500b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9500b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655941023 0 0) 0x7ff95015e2a0 con 0x7ff95015a4e0 2023-12-13T23:57:53.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff93c003520 con 0x7ff95015a4e0 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9561e4640 1 -- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 msgr2=0x7ff9500b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9500b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9561e4640 1 -- 172.21.15.124:0/1737343986 >> v1:172.21.15.124:6789/0 conn(0x7ff95015a4e0 legacy=0x7ff95015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806980172 0 0) 0x7ff93c003520 con 0x7ff95015a4e0 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9561e4640 1 -- 172.21.15.124:0/1737343986 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95015b240 con 0x7ff950143490 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.357+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9500b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.358+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff950143490 0x7ff9500b0cf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0097a0 tx=0x7ff94c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cbb722b399ca8cb server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.358+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9480019f0 con 0x7ff950143490 2023-12-13T23:57:53.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.358+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c02e050 con 0x7ff950143490 2023-12-13T23:57:53.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.358+0000 7ff9551e2640 1 -- 172.21.15.124:0/1737343986 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c030730 con 0x7ff950143490 2023-12-13T23:57:53.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 -- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff950143490 msgr2=0x7ff9500b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff950143490 0x7ff9500b0cf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff94c0097a0 tx=0x7ff94c02b790 comp rx=0 tx=0).stop 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 -- 172.21.15.124:0/1737343986 shutdown_connections 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9500b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 --2- 172.21.15.124:0/1737343986 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff950143490 0x7ff9500b0cf0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 -- 172.21.15.124:0/1737343986 >> 172.21.15.124:0/1737343986 conn(0x7ff95006a4d0 msgr2=0x7ff95006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 -- 172.21.15.124:0/1737343986 shutdown_connections 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.359+0000 7ff958448640 1 -- 172.21.15.124:0/1737343986 wait complete. 2023-12-13T23:57:53.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.360+0000 7ff958448640 1 Processor -- start 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.360+0000 7ff958448640 1 -- start start 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 0x7ff95013c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95015a4e0 0x7ff95013b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9501f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95013d680 con 0x7ff950143490 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff950159ea0 con 0x7ff95015a4e0 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95013cde0 con 0x7ff95015e730 2023-12-13T23:57:53.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 0x7ff95013c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 0x7ff95013c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41368/0 (socket says 172.21.15.124:41368) 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 -- 172.21.15.124:0/1656786563 learned_addr learned my addr 172.21.15.124:0/1656786563 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9569e5640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9501f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95015a4e0 0x7ff95013b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 -- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 msgr2=0x7ff9501f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7ff95015e730 0x7ff9501f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 -- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95015a4e0 msgr2=0x7ff95013b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95015a4e0 0x7ff95013b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff940009400 con 0x7ff950143490 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7ff95015a4e0 0x7ff95013b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff9561e4640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 0x7ff95013c800 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02b7d0 tx=0x7ff94c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c03d890 con 0x7ff950143490 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c03d9f0 con 0x7ff950143490 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c009400 con 0x7ff950143490 2023-12-13T23:57:53.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c03dda0 con 0x7ff950143490 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.361+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95013d3a0 con 0x7ff950143490 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.362+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff95015df30 con 0x7ff950143490 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7ff94c039040 con 0x7ff950143490 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff93406b1c0 0x7ff93406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff94c0ab540 con 0x7ff950143490 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7ff934071020 0x7ff934073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff934073af0 con 0x7ff934071020 2023-12-13T23:57:53.368 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff94c0adc00 con 0x7ff950143490 2023-12-13T23:57:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff93406b1c0 0x7ff93406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.363+0000 7ff9569e5640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7ff934071020 0x7ff934073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.364+0000 7ff9569e5640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7ff934071020 0x7ff934073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.364+0000 7ff9559e3640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff93406b1c0 0x7ff93406d680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff95013c4a0 tx=0x7ff940008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.369 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.364+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== osd.0 v2:172.21.15.124:6816/2614715067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7ff934073af0 con 0x7ff934071020 2023-12-13T23:57:53.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 --> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff95006ade0 con 0x7ff934071020 2023-12-13T23:57:53.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff946ffd640 1 -- 172.21.15.124:0/1656786563 <== osd.0 v2:172.21.15.124:6816/2614715067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff95006ade0 con 0x7ff934071020 2023-12-13T23:57:53.377 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:53.378 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:53.378 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7ff934071020 msgr2=0x7ff934073440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff958448640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:6816/2614715067,v1:172.21.15.124:6818/2614715067] conn(0x7ff934071020 0x7ff934073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff93406b1c0 msgr2=0x7ff93406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.377+0000 7ff958448640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7ff93406b1c0 0x7ff93406d680 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff95013c4a0 tx=0x7ff940008040 comp rx=0 tx=0).stop 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 msgr2=0x7ff95013c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 --2- 172.21.15.124:0/1656786563 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff950143490 0x7ff95013c800 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff94c02b7d0 tx=0x7ff94c0046c0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff9569e5640 1 -- 172.21.15.124:0/1656786563 reap_dead start 2023-12-13T23:57:53.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 shutdown_connections 2023-12-13T23:57:53.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 >> 172.21.15.124:0/1656786563 conn(0x7ff95006a4d0 msgr2=0x7ff950141be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 shutdown_connections 2023-12-13T23:57:53.380 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.378+0000 7ff958448640 1 -- 172.21.15.124:0/1656786563 wait complete. 2023-12-13T23:57:53.389 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-13T23:57:53.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.490+0000 7f97f22ad640 1 Processor -- start 2023-12-13T23:57:53.490 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.490+0000 7f97f22ad640 1 -- start start 2023-12-13T23:57:53.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f97ec143490 0x7f97ec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.492 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 0x7f97ec15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec15b520 con 0x7f97ec143490 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97ec143870 con 0x7f97ec15a4e0 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f22ad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec15e2a0 con 0x7f97ec15e730 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97ebfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f97ec143490 0x7f97ec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97ebfff640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f97ec143490 0x7f97ec0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41372/0 (socket says 172.21.15.124:41372) 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97ebfff640 1 -- 172.21.15.124:0/306491684 learned_addr learned my addr 172.21.15.124:0/306491684 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:53.493 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97eb7fe640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.491+0000 7f97f084a640 1 --2- 172.21.15.124:0/306491684 >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 0x7f97ec15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 -- 172.21.15.124:0/306491684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 msgr2=0x7f97ec0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec0b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 -- 172.21.15.124:0/306491684 >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 msgr2=0x7f97ec15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 --2- 172.21.15.124:0/306491684 >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 0x7f97ec15a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 -- 172.21.15.124:0/306491684 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97ec15b240 con 0x7f97ec143490 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97eb7fe640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.494 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97f084a640 1 --2- 172.21.15.124:0/306491684 >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 0x7f97ec15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97ebfff640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec143490 0x7f97ec0b0cf0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f97ec15eb90 tx=0x7f97dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c304e1fddf153d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97eaffd640 1 -- 172.21.15.124:0/306491684 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc031870 con 0x7f97ec143490 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.492+0000 7f97eaffd640 1 -- 172.21.15.124:0/306491684 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97dc0319d0 con 0x7f97ec143490 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97eaffd640 1 -- 172.21.15.124:0/306491684 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97dc031d60 con 0x7f97ec143490 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 -- 172.21.15.124:0/306491684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec143490 msgr2=0x7f97ec0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec143490 0x7f97ec0b0cf0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f97ec15eb90 tx=0x7f97dc02b910 comp rx=0 tx=0).stop 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 -- 172.21.15.124:0/306491684 shutdown_connections 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 --2- 172.21.15.124:0/306491684 >> v2:172.21.15.203:3300/0 conn(0x7f97ec15a4e0 0x7f97ec15a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 --2- 172.21.15.124:0/306491684 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec143490 0x7f97ec0b0cf0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 -- 172.21.15.124:0/306491684 >> 172.21.15.124:0/306491684 conn(0x7f97ec06a4d0 msgr2=0x7f97ec06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 -- 172.21.15.124:0/306491684 shutdown_connections 2023-12-13T23:57:53.496 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.493+0000 7f97f22ad640 1 -- 172.21.15.124:0/306491684 wait complete. 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.494+0000 7f97f22ad640 1 Processor -- start 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.494+0000 7f97f22ad640 1 -- start start 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 0x7f97ec1592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec13b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec13e920 con 0x7f97ec15a4e0 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec159820 con 0x7f97ec143490 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f22ad640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97ec13c060 con 0x7f97ec15e730 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97eb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97eb7fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41388/0 (socket says 172.21.15.124:41388) 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97eb7fe640 1 -- 172.21.15.124:0/1752452154 learned_addr learned my addr 172.21.15.124:0/1752452154 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97f084a640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec13b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.495+0000 7f97ebfff640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 0x7f97ec1592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 -- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 msgr2=0x7f97ec13b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec13b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 -- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 msgr2=0x7f97ec1592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 0x7f97ec1592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 -- 172.21.15.124:0/1752452154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97dc0094d0 con 0x7f97ec15a4e0 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97f084a640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97ebfff640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 0x7f97ec1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97eb7fe640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c820 tx=0x7f97e000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e0007a30 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e0007b90 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97ec13c340 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e000ce10 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.496+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97ec13eb70 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.497+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97ec066f90 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.497+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f97e00113f0 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.497+0000 7f97e8ff9640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 0x7f97d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.498+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f97e00896e0 con 0x7f97ec15a4e0 2023-12-13T23:57:53.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.498+0000 7f97ebfff640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 0x7f97d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.498+0000 7f97ebfff640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 0x7f97d806d680 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f97dc036000 tx=0x7f97dc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.503 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.502+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f97e00547a0 con 0x7f97ec15a4e0 2023-12-13T23:57:53.669 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.670+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f97ec15aeb0 con 0x7f97d806b1c0 2023-12-13T23:57:53.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.670+0000 7f97e8ff9640 1 -- 172.21.15.124:0/1752452154 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f97dc031cc0 con 0x7f97d806b1c0 2023-12-13T23:57:53.670 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 0 to deep-scrub 2023-12-13T23:57:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 msgr2=0x7f97d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 0x7f97d806d680 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f97dc036000 tx=0x7f97dc03c040 comp rx=0 tx=0).stop 2023-12-13T23:57:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 msgr2=0x7f97ec0a84d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f97e000c820 tx=0x7f97e000cce0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 shutdown_connections 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f97d806b1c0 0x7f97d806d680 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f97ec15e730 0x7f97ec13b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97ec15a4e0 0x7f97ec0a84d0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 --2- 172.21.15.124:0/1752452154 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f97ec143490 0x7f97ec1592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 >> 172.21.15.124:0/1752452154 conn(0x7f97ec06a4d0 msgr2=0x7f97ec0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 shutdown_connections 2023-12-13T23:57:53.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.672+0000 7f97f22ad640 1 -- 172.21.15.124:0/1752452154 wait complete. 2023-12-13T23:57:53.682 INFO:tasks.ceph:Scrubbing osd.1 2023-12-13T23:57:53.683 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:53.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.787+0000 7f68caf61640 1 Processor -- start 2023-12-13T23:57:53.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.787+0000 7f68caf61640 1 -- start start 2023-12-13T23:57:53.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68caf61640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f68c40b1400 0x7f68c415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68caf61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c415e730 0x7f68c415aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68caf61640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c415e2a0 con 0x7f68c4143490 2023-12-13T23:57:53.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68caf61640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c415b520 con 0x7f68c40b1400 2023-12-13T23:57:53.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68caf61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c4143870 con 0x7f68c415e730 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68c3fff640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f68c40b1400 0x7f68c415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68c94fe640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c415e730 0x7f68c415aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68c8cfd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f68c4143490 0x7f68c40b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33216/0 (socket says 172.21.15.124:33216) 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.788+0000 7f68c8cfd640 1 -- 172.21.15.124:0/3197551517 learned_addr learned my addr 172.21.15.124:0/3197551517 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.789+0000 7f68c3fff640 1 -- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c415e730 msgr2=0x7f68c415aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.789+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c415e730 0x7f68c415aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.789+0000 7f68c3fff640 1 -- 172.21.15.124:0/3197551517 >> v1:172.21.15.203:6789/0 conn(0x7f68c4143490 legacy=0x7f68c40b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.789+0000 7f68c3fff640 1 -- 172.21.15.124:0/3197551517 --> v2:172.21.15.203:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c415b240 con 0x7f68c40b1400 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.789+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c40b1400 0x7f68c415a4e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097a0 tx=0x7f68ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adc9d22982e4bbf3 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68c37fe640 1 -- 172.21.15.124:0/3197551517 <== mon.1 v2:172.21.15.203:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac031830 con 0x7f68c40b1400 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68c37fe640 1 -- 172.21.15.124:0/3197551517 <== mon.1 v2:172.21.15.203:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac031990 con 0x7f68c40b1400 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68c37fe640 1 -- 172.21.15.124:0/3197551517 <== mon.1 v2:172.21.15.203:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac031d20 con 0x7f68c40b1400 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 -- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c40b1400 msgr2=0x7f68c415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 --2- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c40b1400 0x7f68c415a4e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0097a0 tx=0x7f68ac02b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 -- 172.21.15.124:0/3197551517 shutdown_connections 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 --2- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c415e730 0x7f68c415aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 --2- 172.21.15.124:0/3197551517 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c40b1400 0x7f68c415a4e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 -- 172.21.15.124:0/3197551517 >> 172.21.15.124:0/3197551517 conn(0x7f68c406a4d0 msgr2=0x7f68c406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 -- 172.21.15.124:0/3197551517 shutdown_connections 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.790+0000 7f68caf61640 1 -- 172.21.15.124:0/3197551517 wait complete. 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.791+0000 7f68caf61640 1 Processor -- start 2023-12-13T23:57:53.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.791+0000 7f68caf61640 1 -- start start 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 0x7f68c41f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c4143490 0x7f68c41e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c415e730 0x7f68c4070600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c41e5390 con 0x7f68c40b1400 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c41e3f50 con 0x7f68c415e730 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68caf61640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40accd0 con 0x7f68c4143490 2023-12-13T23:57:53.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68c8cfd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 0x7f68c41f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68c8cfd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 0x7f68c41f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41398/0 (socket says 172.21.15.124:41398) 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.792+0000 7f68c8cfd640 1 -- 172.21.15.124:0/3660107425 learned_addr learned my addr 172.21.15.124:0/3660107425 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c94fe640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c415e730 0x7f68c4070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c4143490 0x7f68c41e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 -- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c4143490 msgr2=0x7f68c41e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c4143490 0x7f68c41e3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 -- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c415e730 msgr2=0x7f68c4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c415e730 0x7f68c4070600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac009400 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c94fe640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f68c415e730 0x7f68c4070600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f68c4143490 0x7f68c41e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c8cfd640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 0x7f68c41f6880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00cba0 tx=0x7f68bc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00ad40 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc012400 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c41f77b0 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00aea0 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.793+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c41f71f0 con 0x7f68c40b1400 2023-12-13T23:57:53.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.794+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68c415af60 con 0x7f68c40b1400 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.794+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f68bc007070 con 0x7f68c40b1400 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c17fa640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f68a406b1c0 0x7f68a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68bc0895f0 con 0x7f68c40b1400 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c17fa640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f68a4071020 0x7f68a4073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68a4073af0 con 0x7f68a4071020 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68bc08bcb0 con 0x7f68c40b1400 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f68a406b1c0 0x7f68a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c94fe640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f68a4071020 0x7f68a4073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c94fe640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f68a4071020 0x7f68a4073440 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.795+0000 7f68c3fff640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f68a406b1c0 0x7f68a406d680 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f68ac036000 tx=0x7f68ac008030 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.800 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.796+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== osd.1 v2:172.21.15.124:6800/3276151417 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f68a4073af0 con 0x7f68a4071020 2023-12-13T23:57:53.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 --> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f68c415df30 con 0x7f68a4071020 2023-12-13T23:57:53.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68c17fa640 1 -- 172.21.15.124:0/3660107425 <== osd.1 v2:172.21.15.124:6800/3276151417 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f68b4009b00 con 0x7f68a4071020 2023-12-13T23:57:53.809 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:53.809 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:53.809 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:53.810 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f68a4071020 msgr2=0x7f68a4073440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:6800/3276151417,v1:172.21.15.124:6801/3276151417] conn(0x7f68a4071020 0x7f68a4073440 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f68a406b1c0 msgr2=0x7f68a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f68a406b1c0 0x7f68a406d680 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f68ac036000 tx=0x7f68ac008030 comp rx=0 tx=0).stop 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 msgr2=0x7f68c41f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 --2- 172.21.15.124:0/3660107425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f68c40b1400 0x7f68c41f6880 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f68bc00cba0 tx=0x7f68bc00a430 comp rx=0 tx=0).stop 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68c94fe640 1 -- 172.21.15.124:0/3660107425 reap_dead start 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 shutdown_connections 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.809+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 >> 172.21.15.124:0/3660107425 conn(0x7f68c406a4d0 msgr2=0x7f68c40aec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.811 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.810+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 shutdown_connections 2023-12-13T23:57:53.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.810+0000 7f68caf61640 1 -- 172.21.15.124:0/3660107425 wait complete. 2023-12-13T23:57:53.819 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-13T23:57:53.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.919+0000 7f21c7ada640 1 Processor -- start 2023-12-13T23:57:53.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.919+0000 7f21c7ada640 1 -- start start 2023-12-13T23:57:53.921 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c7ada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21c015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c7ada640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c015e2a0 con 0x7f21c0143490 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c7ada640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c015b240 con 0x7f21c015e730 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c7ada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c015b520 con 0x7f21c00b1400 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c5876640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f21c0143490 0x7f21c00b0cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41042/0 (socket says 172.21.15.124:41042) 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.920+0000 7f21c5876640 1 -- 172.21.15.124:0/4089121006 learned_addr learned my addr 172.21.15.124:0/4089121006 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526049198 0 0) 0x7f21c015e2a0 con 0x7f21c0143490 2023-12-13T23:57:53.922 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5075640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21c015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21ac003520 con 0x7f21c0143490 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 927668774 0 0) 0x7f21ac003520 con 0x7f21c0143490 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 msgr2=0x7f21c015a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21c015a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 >> v1:172.21.15.203:6789/0 conn(0x7f21c015e730 legacy=0x7f21c015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c015dc50 con 0x7f21c0143490 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f21bc0019f0 con 0x7f21c0143490 2023-12-13T23:57:53.923 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 >> v1:172.21.15.124:6789/0 conn(0x7f21c0143490 legacy=0x7f21c00b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21ac0039a0 0x7f21ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac004d00 0x7f21ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c015b520 con 0x7f21ac0039a0 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac00ba30 con 0x7f21ac004d00 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21ac00bbf0 con 0x7f21c00b1400 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21ac0039a0 0x7f21ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.924 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c6077640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac004d00 0x7f21ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5075640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 -- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 msgr2=0x7f21ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c00b1400 0x7f21ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 -- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac004d00 msgr2=0x7f21ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac004d00 0x7f21ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.921+0000 7f21c5876640 1 -- 172.21.15.124:0/4089121006 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21ac003520 con 0x7f21ac0039a0 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.922+0000 7f21c5876640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21ac0039a0 0x7f21ac003dc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00a800 tx=0x7f21ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f951b3c6e0a09e66 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.922+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0072d0 con 0x7f21ac0039a0 2023-12-13T23:57:53.925 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.922+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc02f070 con 0x7f21ac0039a0 2023-12-13T23:57:53.926 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.922+0000 7f21c4874640 1 -- 172.21.15.124:0/4089121006 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0076b0 con 0x7f21ac0039a0 2023-12-13T23:57:53.928 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 -- 172.21.15.124:0/4089121006 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21ac0039a0 msgr2=0x7f21ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 --2- 172.21.15.124:0/4089121006 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21ac0039a0 0x7f21ac003dc0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f21bc00a800 tx=0x7f21ac0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c6077640 1 -- 172.21.15.124:0/4089121006 reap_dead start 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 -- 172.21.15.124:0/4089121006 shutdown_connections 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 -- 172.21.15.124:0/4089121006 >> 172.21.15.124:0/4089121006 conn(0x7f21c006a4d0 msgr2=0x7f21c006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 -- 172.21.15.124:0/4089121006 shutdown_connections 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.923+0000 7f21c7ada640 1 -- 172.21.15.124:0/4089121006 wait complete. 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.924+0000 7f21c7ada640 1 Processor -- start 2023-12-13T23:57:53.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.924+0000 7f21c7ada640 1 -- start start 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac0039a0 0x7f21c01f1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c01598d0 0x7f21c00ada90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00a94d0 con 0x7f21c0159d50 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00a97b0 con 0x7f21ac0039a0 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21c00a9a90 con 0x7f21c01598d0 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.930 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41414/0 (socket says 172.21.15.124:41414) 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 -- 172.21.15.124:0/2075097209 learned_addr learned my addr 172.21.15.124:0/2075097209 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 -- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c01598d0 msgr2=0x7f21c00ada90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c01598d0 0x7f21c00ada90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 -- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac0039a0 msgr2=0x7f21c01f1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac0039a0 0x7f21c01f1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 -- 172.21.15.124:0/2075097209 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c00a9d70 con 0x7f21c0159d50 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c6077640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f21c00cef60 tx=0x7f21b800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80053a0 con 0x7f21c0159d50 2023-12-13T23:57:53.931 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b80035f0 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b80039a0 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c00aa050 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.925+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c00adfd0 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.926+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c00aa240 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.927+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f21b8003b00 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.927+0000 7f21b67fc640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 0x7f21a406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.927+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f21b8089350 con 0x7f21c0159d50 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.927+0000 7f21c5876640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 0x7f21a406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:53.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.928+0000 7f21c5876640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 0x7f21a406d680 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f21bc034000 tx=0x7f21bc02f000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:53.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:53.931+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f21b8054380 con 0x7f21c0159d50 2023-12-13T23:57:54.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.098+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f21c0067770 con 0x7f21a406b1c0 2023-12-13T23:57:54.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.099+0000 7f21b67fc640 1 -- 172.21.15.124:0/2075097209 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f21c0067770 con 0x7f21a406b1c0 2023-12-13T23:57:54.098 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 1 to deep-scrub 2023-12-13T23:57:54.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 msgr2=0x7f21a406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 0x7f21a406d680 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f21bc034000 tx=0x7f21bc02f000 comp rx=0 tx=0).stop 2023-12-13T23:57:54.100 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 msgr2=0x7f21c00aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f21c00cef60 tx=0x7f21b800f920 comp rx=0 tx=0).stop 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 shutdown_connections 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f21a406b1c0 0x7f21a406d680 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21c0159d50 0x7f21c00aba70 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f21c01598d0 0x7f21c00ada90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 --2- 172.21.15.124:0/2075097209 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f21ac0039a0 0x7f21c01f1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 >> 172.21.15.124:0/2075097209 conn(0x7f21c006a4d0 msgr2=0x7f21c0070390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.101 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 shutdown_connections 2023-12-13T23:57:54.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.101+0000 7f21c7ada640 1 -- 172.21.15.124:0/2075097209 wait complete. 2023-12-13T23:57:54.111 INFO:tasks.ceph:Scrubbing osd.2 2023-12-13T23:57:54.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:54.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.216+0000 7f278e487640 1 Processor -- start 2023-12-13T23:57:54.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.216+0000 7f278e487640 1 -- start start 2023-12-13T23:57:54.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278e487640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278e487640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278e487640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278815e2a0 con 0x7f278815e730 2023-12-13T23:57:54.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278e487640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278815b520 con 0x7f278815abf0 2023-12-13T23:57:54.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278e487640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2788143870 con 0x7f2788143490 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278ca24640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f27877fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f278815e730 0x7f278815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33220/0 (socket says 172.21.15.124:33220) 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3301/0 says I am v2:172.21.15.124:53454/0 (socket says 172.21.15.124:53454) 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278ca24640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.203:3300/0 says I am v2:172.21.15.124:33276/0 (socket says 172.21.15.124:33276) 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f27877fe640 1 -- 172.21.15.124:0/4257196517 learned_addr learned my addr 172.21.15.124:0/4257196517 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:54.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 -- 172.21.15.124:0/4257196517 >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 msgr2=0x7f27880b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 --2- 172.21.15.124:0/4257196517 >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 -- 172.21.15.124:0/4257196517 >> v1:172.21.15.203:6789/0 conn(0x7f278815e730 legacy=0x7f278815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f2787fff640 1 -- 172.21.15.124:0/4257196517 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278815b240 con 0x7f2788143490 2023-12-13T23:57:54.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.217+0000 7f278ca24640 1 --2- 172.21.15.124:0/4257196517 >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:54.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.218+0000 7f2787fff640 1 --2- 172.21.15.124:0/4257196517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27780097a0 tx=0x7f277802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c4ceb6a4ab55408 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.218+0000 7f2786ffd640 1 -- 172.21.15.124:0/4257196517 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778031830 con 0x7f2788143490 2023-12-13T23:57:54.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.218+0000 7f2786ffd640 1 -- 172.21.15.124:0/4257196517 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2778031990 con 0x7f2788143490 2023-12-13T23:57:54.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.218+0000 7f2786ffd640 1 -- 172.21.15.124:0/4257196517 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778031d20 con 0x7f2788143490 2023-12-13T23:57:54.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 -- 172.21.15.124:0/4257196517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 msgr2=0x7f27880b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 --2- 172.21.15.124:0/4257196517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f27780097a0 tx=0x7f277802b8d0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 -- 172.21.15.124:0/4257196517 shutdown_connections 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 --2- 172.21.15.124:0/4257196517 >> v2:172.21.15.203:3300/0 conn(0x7f278815abf0 0x7f27880b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 --2- 172.21.15.124:0/4257196517 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2788143490 0x7f27880b0cf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 -- 172.21.15.124:0/4257196517 >> 172.21.15.124:0/4257196517 conn(0x7f278806a4d0 msgr2=0x7f278806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 -- 172.21.15.124:0/4257196517 shutdown_connections 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.219+0000 7f278e487640 1 -- 172.21.15.124:0/4257196517 wait complete. 2023-12-13T23:57:54.225 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.220+0000 7f278e487640 1 Processor -- start 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.220+0000 7f278e487640 1 -- start start 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2788143490 0x7f2788158d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 0x7f2788163020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f278815e730 0x7f2788163560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278813b960 con 0x7f278815abf0 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27881592b0 con 0x7f2788143490 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278e487640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f278813b680 con 0x7f278815e730 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 0x7f2788163020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.226 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 0x7f2788163020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41428/0 (socket says 172.21.15.124:41428) 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 -- 172.21.15.124:0/1687220124 learned_addr learned my addr 172.21.15.124:0/1687220124 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f2787fff640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2788143490 0x7f2788158d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f278ca24640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f278815e730 0x7f2788163560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 -- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f278815e730 msgr2=0x7f2788163560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f278815e730 0x7f2788163560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 -- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2788143490 msgr2=0x7f2788158d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.221+0000 7f27877fe640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2788143490 0x7f2788158d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f27877fe640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780009a70 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f27877fe640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 0x7f2788163020 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c480 tx=0x7f277c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c005ce0 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277c005e40 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277c0072d0 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2778009400 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.222+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278813d060 con 0x7f278815abf0 2023-12-13T23:57:54.228 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.223+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f278815df30 con 0x7f278815abf0 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.223+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f277c0074f0 con 0x7f278815abf0 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2784ff9640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f277406b1c0 0x7f277406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f277c089330 con 0x7f278815abf0 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2784ff9640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7f2774071020 0x7f2774073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2774073af0 con 0x7f2774071020 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f277c089780 con 0x7f278815abf0 2023-12-13T23:57:54.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2787fff640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f277406b1c0 0x7f277406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f278ca24640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7f2774071020 0x7f2774073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f278ca24640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7f2774071020 0x7f2774073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.224+0000 7f2787fff640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f277406b1c0 0x7f277406d680 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2778036000 tx=0x7f277803c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.230 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.225+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== osd.2 v2:172.21.15.124:6808/1670235117 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f2774073af0 con 0x7f2774071020 2023-12-13T23:57:54.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.237+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 --> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f278815b000 con 0x7f2774071020 2023-12-13T23:57:54.237 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f2784ff9640 1 -- 172.21.15.124:0/1687220124 <== osd.2 v2:172.21.15.124:6808/1670235117 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f278000f040 con 0x7f2774071020 2023-12-13T23:57:54.237 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:54.238 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:54.238 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:54.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7f2774071020 msgr2=0x7f2774073440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.239 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:6808/1670235117,v1:172.21.15.124:6809/1670235117] conn(0x7f2774071020 0x7f2774073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f277406b1c0 msgr2=0x7f277406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f277406b1c0 0x7f277406d680 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2778036000 tx=0x7f277803c040 comp rx=0 tx=0).stop 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 msgr2=0x7f2788163020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 --2- 172.21.15.124:0/1687220124 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f278815abf0 0x7f2788163020 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f277c00c480 tx=0x7f277c00c880 comp rx=0 tx=0).stop 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278ca24640 1 -- 172.21.15.124:0/1687220124 reap_dead start 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 shutdown_connections 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 >> 172.21.15.124:0/1687220124 conn(0x7f278806a4d0 msgr2=0x7f27880719c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.240 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 shutdown_connections 2023-12-13T23:57:54.241 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.238+0000 7f278e487640 1 -- 172.21.15.124:0/1687220124 wait complete. 2023-12-13T23:57:54.248 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-13T23:57:54.348 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.349+0000 7fe1fb190640 1 Processor -- start 2023-12-13T23:57:54.349 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.349+0000 7fe1fb190640 1 -- start start 2023-12-13T23:57:54.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1fb190640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe1f4143490 0x7fe1f40b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.351 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1fb190640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f40b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1fb190640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1f415e2a0 con 0x7fe1f415a4e0 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1fb190640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1f415b520 con 0x7fe1f4143490 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1fb190640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4143870 con 0x7fe1f415e730 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe1f4143490 0x7fe1f40b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe1f4143490 0x7fe1f40b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41440/0 (socket says 172.21.15.124:41440) 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/699776469 learned_addr learned my addr 172.21.15.124:0/699776469 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:54.352 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f3fff640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f40b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/699776469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 msgr2=0x7fe1f40b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f40b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/699776469 >> v1:172.21.15.203:6789/0 conn(0x7fe1f415a4e0 legacy=0x7fe1f415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/699776469 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f415b240 con 0x7fe1f4143490 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.350+0000 7fe1f3fff640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1f8f2c640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f40b0cf0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009870 tx=0x7fe1e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed029948d76c986 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.353 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1f37fe640 1 -- 172.21.15.124:0/699776469 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8030840 con 0x7fe1f4143490 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1f37fe640 1 -- 172.21.15.124:0/699776469 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e80309a0 con 0x7fe1f4143490 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1f37fe640 1 -- 172.21.15.124:0/699776469 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8030d30 con 0x7fe1f4143490 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1fb190640 1 -- 172.21.15.124:0/699776469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 msgr2=0x7fe1f40b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.351+0000 7fe1fb190640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f40b0cf0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009870 tx=0x7fe1e802b980 comp rx=0 tx=0).stop 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 -- 172.21.15.124:0/699776469 shutdown_connections 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f40b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 --2- 172.21.15.124:0/699776469 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f40b0cf0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 -- 172.21.15.124:0/699776469 >> 172.21.15.124:0/699776469 conn(0x7fe1f406a4d0 msgr2=0x7fe1f406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 -- 172.21.15.124:0/699776469 shutdown_connections 2023-12-13T23:57:54.355 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.352+0000 7fe1fb190640 1 -- 172.21.15.124:0/699776469 wait complete. 2023-12-13T23:57:54.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.353+0000 7fe1fb190640 1 Processor -- start 2023-12-13T23:57:54.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.353+0000 7fe1fb190640 1 -- start start 2023-12-13T23:57:54.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f415a4e0 0x7fe1f413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f41423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f41f1450 con 0x7fe1f4143490 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f413e290 con 0x7fe1f415a4e0 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f4139680 con 0x7fe1f415e730 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41442/0 (socket says 172.21.15.124:41442) 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/1534085032 learned_addr learned my addr 172.21.15.124:0/1534085032 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:54.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f972d640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f41423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f3fff640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f415a4e0 0x7fe1f413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 msgr2=0x7fe1f41423a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f41423a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f415a4e0 msgr2=0x7fe1f413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f415a4e0 0x7fe1f413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 -- 172.21.15.124:0/1534085032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1e80094d0 con 0x7fe1f4143490 2023-12-13T23:57:54.360 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f8f2c640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009870 tx=0x7fe1e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8039c90 con 0x7fe1f4143490 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1e8039df0 con 0x7fe1f4143490 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f4139960 con 0x7fe1f4143490 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8043620 con 0x7fe1f4143490 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.354+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f4139f20 con 0x7fe1f4143490 2023-12-13T23:57:54.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.355+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f4069400 con 0x7fe1f4143490 2023-12-13T23:57:54.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.356+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fe1e8043780 con 0x7fe1f4143490 2023-12-13T23:57:54.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.356+0000 7fe1f17fa640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 0x7fe1d806d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.356+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe1e803e070 con 0x7fe1f4143490 2023-12-13T23:57:54.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.356+0000 7fe1f3fff640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 0x7fe1d806d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.357+0000 7fe1f3fff640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 0x7fe1d806d680 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1f413bfc0 tx=0x7fe1e0008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.360+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fe1e8077870 con 0x7fe1f4143490 2023-12-13T23:57:54.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.528+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1f415df30 con 0x7fe1d806b1c0 2023-12-13T23:57:54.528 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.528+0000 7fe1f17fa640 1 -- 172.21.15.124:0/1534085032 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe1e000b140 con 0x7fe1d806b1c0 2023-12-13T23:57:54.528 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 2 to deep-scrub 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 msgr2=0x7fe1d806d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 0x7fe1d806d680 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1f413bfc0 tx=0x7fe1e0008040 comp rx=0 tx=0).stop 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 msgr2=0x7fe1f41599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8009870 tx=0x7fe1e8004730 comp rx=0 tx=0).stop 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 shutdown_connections 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fe1d806b1c0 0x7fe1d806d680 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.530 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fe1f415a4e0 0x7fe1f413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fe1f415e730 0x7fe1f41423a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 --2- 172.21.15.124:0/1534085032 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe1f4143490 0x7fe1f41599f0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 >> 172.21.15.124:0/1534085032 conn(0x7fe1f406a4d0 msgr2=0x7fe1f4071cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 shutdown_connections 2023-12-13T23:57:54.531 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.530+0000 7fe1fb190640 1 -- 172.21.15.124:0/1534085032 wait complete. 2023-12-13T23:57:54.540 INFO:tasks.ceph:Scrubbing osd.3 2023-12-13T23:57:54.541 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:54.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.645+0000 7f7204701640 1 Processor -- start 2023-12-13T23:57:54.645 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.645+0000 7f7204701640 1 -- start start 2023-12-13T23:57:54.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7204701640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 0x7f71fc0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7204701640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 0x7f71fc0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7204701640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71fc0b49f0 con 0x7f71fc0b3390 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7204701640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71fc0b3de0 con 0x7f71fc150560 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7204701640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc0b40c0 con 0x7f71fc150940 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f720249d640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 0x7f71fc0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7202c9e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 0x7f71fc0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7201c9c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f71fc0b3390 0x7f71fc15d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.124:33232/0 (socket says 172.21.15.124:33232) 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.646+0000 7f7201c9c640 1 -- 172.21.15.124:0/1373055604 learned_addr learned my addr 172.21.15.124:0/1373055604 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:54.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f720149b640 1 -- 172.21.15.124:0/1373055604 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158520993 0 0) 0x7f71fc0b49f0 con 0x7f71fc0b3390 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f720149b640 1 -- 172.21.15.124:0/1373055604 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71e8003520 con 0x7f71fc0b3390 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f7202c9e640 1 -- 172.21.15.124:0/1373055604 >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 msgr2=0x7f71fc0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f7202c9e640 1 --2- 172.21.15.124:0/1373055604 >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 0x7f71fc0b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f7202c9e640 1 -- 172.21.15.124:0/1373055604 >> v1:172.21.15.203:6789/0 conn(0x7f71fc0b3390 legacy=0x7f71fc15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f7202c9e640 1 -- 172.21.15.124:0/1373055604 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71fc0b43a0 con 0x7f71fc150940 2023-12-13T23:57:54.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.647+0000 7f720249d640 1 --2- 172.21.15.124:0/1373055604 >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 0x7f71fc0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7202c9e640 1 --2- 172.21.15.124:0/1373055604 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 0x7f71fc0ce0e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71f400a940 tx=0x7f71f402cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7c7a7d84a9a51f60 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f720149b640 1 -- 172.21.15.124:0/1373055604 <== mon.2 v2:172.21.15.203:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f402db30 con 0x7f71fc150940 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f720149b640 1 -- 172.21.15.124:0/1373055604 <== mon.2 v2:172.21.15.203:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f402dc90 con 0x7f71fc150940 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f720149b640 1 -- 172.21.15.124:0/1373055604 <== mon.2 v2:172.21.15.203:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4035440 con 0x7f71fc150940 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 -- 172.21.15.124:0/1373055604 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 msgr2=0x7f71fc0ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.653 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 --2- 172.21.15.124:0/1373055604 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 0x7f71fc0ce0e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f71f400a940 tx=0x7f71f402cd80 comp rx=0 tx=0).stop 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 -- 172.21.15.124:0/1373055604 shutdown_connections 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 --2- 172.21.15.124:0/1373055604 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150940 0x7f71fc0ce0e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 --2- 172.21.15.124:0/1373055604 >> v2:172.21.15.203:3300/0 conn(0x7f71fc150560 0x7f71fc0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 -- 172.21.15.124:0/1373055604 >> 172.21.15.124:0/1373055604 conn(0x7f71fc06a4d0 msgr2=0x7f71fc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 -- 172.21.15.124:0/1373055604 shutdown_connections 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.648+0000 7f7204701640 1 -- 172.21.15.124:0/1373055604 wait complete. 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 Processor -- start 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 -- start start 2023-12-13T23:57:54.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 0x7f71fc153a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150560 0x7f71fc148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71fc150d30 0x7f71fc154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc158ec0 con 0x7f71fc0b3390 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc149290 con 0x7f71fc150d30 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f7204701640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc158560 con 0x7f71fc150560 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f720249d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 0x7f71fc153a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f720249d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 0x7f71fc153a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41458/0 (socket says 172.21.15.124:41458) 2023-12-13T23:57:54.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.650+0000 7f720249d640 1 -- 172.21.15.124:0/2494242708 learned_addr learned my addr 172.21.15.124:0/2494242708 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f7201c9c640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150560 0x7f71fc148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f7202c9e640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71fc150d30 0x7f71fc154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 -- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150560 msgr2=0x7f71fc148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150560 0x7f71fc148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 -- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71fc150d30 msgr2=0x7f71fc154a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f71fc150d30 0x7f71fc154a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009a70 con 0x7f71fc0b3390 2023-12-13T23:57:54.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f7201c9c640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f71fc150560 0x7f71fc148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:54.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f720249d640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 0x7f71fc153a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009f60 tx=0x7f71f8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8014680 con 0x7f71fc0b3390 2023-12-13T23:57:54.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f800f040 con 0x7f71fc0b3390 2023-12-13T23:57:54.657 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f400a5a0 con 0x7f71fc0b3390 2023-12-13T23:57:54.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8014a00 con 0x7f71fc0b3390 2023-12-13T23:57:54.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.651+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc158800 con 0x7f71fc0b3390 2023-12-13T23:57:54.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.652+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f71fc15dec0 con 0x7f71fc0b3390 2023-12-13T23:57:54.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.652+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f71f8014b60 con 0x7f71fc0b3390 2023-12-13T23:57:54.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f71f2ffd640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71e006b1c0 0x7f71e006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f71f808b100 con 0x7f71fc0b3390 2023-12-13T23:57:54.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f71f2ffd640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f71e0071020 0x7f71e0073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71e0073af0 con 0x7f71e0071020 2023-12-13T23:57:54.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71f808b550 con 0x7f71fc0b3390 2023-12-13T23:57:54.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f7201c9c640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71e006b1c0 0x7f71e006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f7202c9e640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f71e0071020 0x7f71e0073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f7202c9e640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f71e0071020 0x7f71e0073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.653+0000 7f7201c9c640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71e006b1c0 0x7f71e006d680 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f71fc1564d0 tx=0x7f71ec009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.654+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== osd.3 v2:172.21.15.124:6817/3893379194 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f71e0073af0 con 0x7f71e0071020 2023-12-13T23:57:54.666 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 --> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f71fc159130 con 0x7f71e0071020 2023-12-13T23:57:54.667 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f71f2ffd640 1 -- 172.21.15.124:0/2494242708 <== osd.3 v2:172.21.15.124:6817/3893379194 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f71fc159130 con 0x7f71e0071020 2023-12-13T23:57:54.667 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:54.668 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:54.668 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:54.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f71e0071020 msgr2=0x7f71e0073440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:6817/3893379194,v1:172.21.15.124:6819/3893379194] conn(0x7f71e0071020 0x7f71e0073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71e006b1c0 msgr2=0x7f71e006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.670 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f71e006b1c0 0x7f71e006d680 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f71fc1564d0 tx=0x7f71ec009040 comp rx=0 tx=0).stop 2023-12-13T23:57:54.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 msgr2=0x7f71fc153a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 --2- 172.21.15.124:0/2494242708 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f71fc0b3390 0x7f71fc153a00 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009f60 tx=0x7f71f8009f90 comp rx=0 tx=0).stop 2023-12-13T23:57:54.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7202c9e640 1 -- 172.21.15.124:0/2494242708 reap_dead start 2023-12-13T23:57:54.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 shutdown_connections 2023-12-13T23:57:54.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 >> 172.21.15.124:0/2494242708 conn(0x7f71fc06a4d0 msgr2=0x7f71fc0aecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 shutdown_connections 2023-12-13T23:57:54.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.667+0000 7f7204701640 1 -- 172.21.15.124:0/2494242708 wait complete. 2023-12-13T23:57:54.677 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-13T23:57:54.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.780+0000 7f5169978640 1 Processor -- start 2023-12-13T23:57:54.780 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.780+0000 7f5169978640 1 -- start start 2023-12-13T23:57:54.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5169978640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f51640b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5169978640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f51640d2d90 0x7f51640b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5169978640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51640b2b10 con 0x7f51640d2910 2023-12-13T23:57:54.784 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5169978640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51640d1590 con 0x7f51640d2d90 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5169978640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640d1870 con 0x7f516415d330 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f51627fc640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f51640d2910 0x7f51640d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41058/0 (socket says 172.21.15.124:41058) 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f51627fc640 1 -- 172.21.15.124:0/986548804 learned_addr learned my addr 172.21.15.124:0/986548804 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5162ffd640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f51640b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f51637fe640 1 --2- 172.21.15.124:0/986548804 >> v2:172.21.15.203:3300/0 conn(0x7f51640d2d90 0x7f51640b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473810112 0 0) 0x7f51640b2b10 con 0x7f51640d2910 2023-12-13T23:57:54.785 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f514c003520 con 0x7f51640d2910 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634779715 0 0) 0x7f514c003520 con 0x7f51640d2910 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 msgr2=0x7f51640b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f51640b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 >> v2:172.21.15.203:3300/0 conn(0x7f51640d2d90 msgr2=0x7f51640b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 --2- 172.21.15.124:0/986548804 >> v2:172.21.15.203:3300/0 conn(0x7f51640d2d90 0x7f51640b6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640d1b50 con 0x7f51640d2910 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.781+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f5158001670 con 0x7f51640d2910 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 >> v1:172.21.15.124:6789/0 conn(0x7f51640d2910 legacy=0x7f51640d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f514c003970 0x7f514c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f514c0047c0 0x7f514c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f514c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640d1590 con 0x7f514c0047c0 2023-12-13T23:57:54.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640d1870 con 0x7f514c003970 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c00b980 con 0x7f516415d330 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f514c0047c0 0x7f514c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f5162ffd640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f514c003970 0x7f514c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51637fe640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f514c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 -- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 msgr2=0x7f514c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f514c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 -- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f514c003970 msgr2=0x7f514c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:54.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f514c003970 0x7f514c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51627fc640 1 -- 172.21.15.124:0/986548804 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c003520 con 0x7f514c0047c0 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.782+0000 7f51637fe640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f516415d330 0x7f514c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f51627fc640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f514c0047c0 0x7f514c004c40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f515800c270 tx=0x7f514c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e85aaa238f4db6f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515802cae0 con 0x7f514c0047c0 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515802cc40 con 0x7f514c0047c0 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5161ffb640 1 -- 172.21.15.124:0/986548804 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51580058d0 con 0x7f514c0047c0 2023-12-13T23:57:54.789 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 -- 172.21.15.124:0/986548804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f514c0047c0 msgr2=0x7f514c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 --2- 172.21.15.124:0/986548804 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f514c0047c0 0x7f514c004c40 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f515800c270 tx=0x7f514c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f51637fe640 1 -- 172.21.15.124:0/986548804 reap_dead start 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 -- 172.21.15.124:0/986548804 shutdown_connections 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 -- 172.21.15.124:0/986548804 >> 172.21.15.124:0/986548804 conn(0x7f5164072020 msgr2=0x7f5164072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 -- 172.21.15.124:0/986548804 shutdown_connections 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.783+0000 7f5169978640 1 -- 172.21.15.124:0/986548804 wait complete. 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.784+0000 7f5169978640 1 Processor -- start 2023-12-13T23:57:54.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 -- start start 2023-12-13T23:57:54.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 0x7f51640c9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 0x7f516414e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.794 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51640cb5e0 con 0x7f51640cc9c0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f516414ec60 con 0x7f51641576e0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f5169978640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51640cbf70 con 0x7f514c0047c0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f51637fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f51637fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41476/0 (socket says 172.21.15.124:41476) 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.785+0000 7f51637fe640 1 -- 172.21.15.124:0/802971522 learned_addr learned my addr 172.21.15.124:0/802971522 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51627fc640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 0x7f516414e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f5162ffd640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 0x7f51640c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 -- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 msgr2=0x7f51640c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 0x7f51640c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 -- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 msgr2=0x7f516414e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 0x7f516414e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 -- 172.21.15.124:0/802971522 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51640cb7c0 con 0x7f51640cc9c0 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51627fc640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 0x7f516414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f5162ffd640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 0x7f51640c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f51637fe640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5164160e00 tx=0x7f515c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c00e050 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f515c00f040 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51640cbaa0 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515c003990 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.786+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51640cc4a0 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.787+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51640cc6c0 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.787+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f515c015440 con 0x7f51640cc9c0 2023-12-13T23:57:54.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.788+0000 7f514bfff640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 0x7f515006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:54.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.788+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f515c089c70 con 0x7f51640cc9c0 2023-12-13T23:57:54.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.788+0000 7f5162ffd640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 0x7f515006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:54.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.789+0000 7f5162ffd640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 0x7f515006d680 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f51641f6540 tx=0x7f5154008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:54.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.792+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f515c054d30 con 0x7f51640cc9c0 2023-12-13T23:57:54.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.957+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f51640c9ff0 con 0x7f515006b1c0 2023-12-13T23:57:54.957 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.958+0000 7f514bfff640 1 -- 172.21.15.124:0/802971522 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f51640c9ff0 con 0x7f515006b1c0 2023-12-13T23:57:54.958 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 3 to deep-scrub 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 msgr2=0x7f515006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 0x7f515006d680 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f51641f6540 tx=0x7f5154008040 comp rx=0 tx=0).stop 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 msgr2=0x7f516415ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5164160e00 tx=0x7f515c009f90 comp rx=0 tx=0).stop 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 shutdown_connections 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f515006b1c0 0x7f515006d680 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.960 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f51641576e0 0x7f516414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51640cc9c0 0x7f516415ea10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 --2- 172.21.15.124:0/802971522 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f514c0047c0 0x7f51640c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:54.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 >> 172.21.15.124:0/802971522 conn(0x7f5164072020 msgr2=0x7f51640b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:54.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 shutdown_connections 2023-12-13T23:57:54.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:54.960+0000 7f5169978640 1 -- 172.21.15.124:0/802971522 wait complete. 2023-12-13T23:57:54.970 INFO:tasks.ceph:Scrubbing osd.4 2023-12-13T23:57:54.970 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:55.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 Processor -- start 2023-12-13T23:57:55.074 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 -- start start 2023-12-13T23:57:55.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fec8c0ae480 0x7fec8c06d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec8c072390 0x7fec8c06d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c0ae910 con 0x7fec8c15d2b0 2023-12-13T23:57:55.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8c0b2200 con 0x7fec8c072390 2023-12-13T23:57:55.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec92e9b640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8c0b24e0 con 0x7fec8c0ae480 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.075+0000 7fec91438640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec8c072390 0x7fec8c06d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fec8c072390 0x7fec8c06d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41478/0 (socket says 172.21.15.124:41478) 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 -- 172.21.15.124:0/74854601 learned_addr learned my addr 172.21.15.124:0/74854601 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec8bfff640 1 --2- 172.21.15.124:0/74854601 >> v2:172.21.15.203:3300/0 conn(0x7fec8c0ae480 0x7fec8c06d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 -- 172.21.15.124:0/74854601 >> v2:172.21.15.203:3300/0 conn(0x7fec8c0ae480 msgr2=0x7fec8c06d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 --2- 172.21.15.124:0/74854601 >> v2:172.21.15.203:3300/0 conn(0x7fec8c0ae480 0x7fec8c06d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 -- 172.21.15.124:0/74854601 >> v1:172.21.15.203:6789/0 conn(0x7fec8c15d2b0 legacy=0x7fec8c071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.076+0000 7fec91438640 1 -- 172.21.15.124:0/74854601 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0b40a0 con 0x7fec8c072390 2023-12-13T23:57:55.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec91438640 1 --2- 172.21.15.124:0/74854601 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c072390 0x7fec8c06d880 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fec8c160e00 tx=0x7fec8402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c525cefad5db5e9f server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec8b7fe640 1 -- 172.21.15.124:0/74854601 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8402c9f0 con 0x7fec8c072390 2023-12-13T23:57:55.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec8b7fe640 1 -- 172.21.15.124:0/74854601 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8402cb50 con 0x7fec8c072390 2023-12-13T23:57:55.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec8b7fe640 1 -- 172.21.15.124:0/74854601 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec840343f0 con 0x7fec8c072390 2023-12-13T23:57:55.080 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 -- 172.21.15.124:0/74854601 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c072390 msgr2=0x7fec8c06d880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 --2- 172.21.15.124:0/74854601 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c072390 0x7fec8c06d880 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fec8c160e00 tx=0x7fec8402be50 comp rx=0 tx=0).stop 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 -- 172.21.15.124:0/74854601 shutdown_connections 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 --2- 172.21.15.124:0/74854601 >> v2:172.21.15.203:3300/0 conn(0x7fec8c0ae480 0x7fec8c06d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 --2- 172.21.15.124:0/74854601 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c072390 0x7fec8c06d880 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 -- 172.21.15.124:0/74854601 >> 172.21.15.124:0/74854601 conn(0x7fec8c0b4720 msgr2=0x7fec8c0b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 -- 172.21.15.124:0/74854601 shutdown_connections 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.077+0000 7fec92e9b640 1 -- 172.21.15.124:0/74854601 wait complete. 2023-12-13T23:57:55.081 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.078+0000 7fec92e9b640 1 Processor -- start 2023-12-13T23:57:55.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.078+0000 7fec92e9b640 1 -- start start 2023-12-13T23:57:55.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec8c072390 0x7fec8c154920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec8c15d2b0 0x7fec8c0afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 0x7fec8c06c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.082 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c159350 con 0x7fec8c0b0400 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0b0f40 con 0x7fec8c072390 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec92e9b640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0b0980 con 0x7fec8c15d2b0 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec91438640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 0x7fec8c06c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec91438640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 0x7fec8c06c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41494/0 (socket says 172.21.15.124:41494) 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.079+0000 7fec91438640 1 -- 172.21.15.124:0/2899816836 learned_addr learned my addr 172.21.15.124:0/2899816836 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec90c37640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec8c072390 0x7fec8c154920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec8bfff640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec8c15d2b0 0x7fec8c0afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.083 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 -- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec8c15d2b0 msgr2=0x7fec8c0afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec8c15d2b0 0x7fec8c0afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 -- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec8c072390 msgr2=0x7fec8c154920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fec8c072390 0x7fec8c154920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84009b30 con 0x7fec8c0b0400 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec8bfff640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fec8c15d2b0 0x7fec8c0afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec91438640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 0x7fec8c06c930 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fec84036040 tx=0x7fec84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8403e450 con 0x7fec8c0b0400 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec8403e5d0 con 0x7fec8c0b0400 2023-12-13T23:57:55.084 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8c0b2af0 con 0x7fec8c0b0400 2023-12-13T23:57:55.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8403e980 con 0x7fec8c0b0400 2023-12-13T23:57:55.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.080+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c158f70 con 0x7fec8c0b0400 2023-12-13T23:57:55.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.081+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec8c0b2970 con 0x7fec8c0b0400 2023-12-13T23:57:55.085 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.081+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fec8403eae0 con 0x7fec8c0b0400 2023-12-13T23:57:55.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec897fa640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec7006b1f0 0x7fec7006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fec840ac510 con 0x7fec8c0b0400 2023-12-13T23:57:55.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec897fa640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fec70071050 0x7fec70073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.086 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec70073b20 con 0x7fec70071050 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec840aebd0 con 0x7fec8c0b0400 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec90c37640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec7006b1f0 0x7fec7006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec8bfff640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fec70071050 0x7fec70073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.082+0000 7fec90c37640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec7006b1f0 0x7fec7006d6b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.083+0000 7fec8bfff640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fec70071050 0x7fec70073470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.087 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.083+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== osd.4 v2:172.21.15.203:6809/3737544324 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fec70073b20 con 0x7fec70071050 2023-12-13T23:57:55.095 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.096+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 --> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec8c15d690 con 0x7fec70071050 2023-12-13T23:57:55.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec897fa640 1 -- 172.21.15.124:0/2899816836 <== osd.4 v2:172.21.15.203:6809/3737544324 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fec7800c040 con 0x7fec70071050 2023-12-13T23:57:55.096 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:55.096 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:55.097 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fec70071050 msgr2=0x7fec70073470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6809/3737544324,v1:172.21.15.203:6811/3737544324] conn(0x7fec70071050 0x7fec70073470 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec7006b1f0 msgr2=0x7fec7006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fec7006b1f0 0x7fec7006d6b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fec80006fd0 tx=0x7fec80008040 comp rx=0 tx=0).stop 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 msgr2=0x7fec8c06c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 --2- 172.21.15.124:0/2899816836 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec8c0b0400 0x7fec8c06c930 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fec84036040 tx=0x7fec84004a40 comp rx=0 tx=0).stop 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec91438640 1 -- 172.21.15.124:0/2899816836 reap_dead start 2023-12-13T23:57:55.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 shutdown_connections 2023-12-13T23:57:55.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 >> 172.21.15.124:0/2899816836 conn(0x7fec8c0b4720 msgr2=0x7fec8c158050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 shutdown_connections 2023-12-13T23:57:55.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.097+0000 7fec92e9b640 1 -- 172.21.15.124:0/2899816836 wait complete. 2023-12-13T23:57:55.107 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-13T23:57:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.207+0000 7f0787017640 1 Processor -- start 2023-12-13T23:57:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.207+0000 7f0787017640 1 -- start start 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0787017640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f078015d330 0x7f07800b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0787017640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f07800d3900 0x7f07800b6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0787017640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07800b2b10 con 0x7f07800d2910 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0787017640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07800d1590 con 0x7f078015d330 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0787017640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07800d1870 con 0x7f07800d3900 2023-12-13T23:57:55.209 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f0784db3640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f078015d330 0x7f07800b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f077ffff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f07800d2910 0x7f07800d31f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41062/0 (socket says 172.21.15.124:41062) 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.208+0000 7f077ffff640 1 -- 172.21.15.124:0/1020133334 learned_addr learned my addr 172.21.15.124:0/1020133334 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f0784db3640 1 -- 172.21.15.124:0/1020133334 >> v2:172.21.15.203:3300/0 conn(0x7f07800d3900 msgr2=0x7f07800b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f0784db3640 1 --2- 172.21.15.124:0/1020133334 >> v2:172.21.15.203:3300/0 conn(0x7f07800d3900 0x7f07800b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f0784db3640 1 -- 172.21.15.124:0/1020133334 >> v1:172.21.15.124:6789/0 conn(0x7f07800d2910 legacy=0x7f07800d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f0784db3640 1 -- 172.21.15.124:0/1020133334 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07800d1b50 con 0x7f078015d330 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f077f7fe640 1 -- 172.21.15.124:0/1020133334 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498397896 0 0) 0x7f07800b2b10 con 0x7f07800d2910 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f0784db3640 1 --2- 172.21.15.124:0/1020133334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f078015d330 0x7f07800b8b80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f078006c500 tx=0x7f077402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b785476440c935 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.209+0000 7f077f7fe640 1 -- 172.21.15.124:0/1020133334 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077402e070 con 0x7f078015d330 2023-12-13T23:57:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f077f7fe640 1 -- 172.21.15.124:0/1020133334 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07740315a0 con 0x7f078015d330 2023-12-13T23:57:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f077f7fe640 1 -- 172.21.15.124:0/1020133334 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774031930 con 0x7f078015d330 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 -- 172.21.15.124:0/1020133334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f078015d330 msgr2=0x7f07800b8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 --2- 172.21.15.124:0/1020133334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f078015d330 0x7f07800b8b80 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f078006c500 tx=0x7f077402b840 comp rx=0 tx=0).stop 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 -- 172.21.15.124:0/1020133334 shutdown_connections 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 --2- 172.21.15.124:0/1020133334 >> v2:172.21.15.203:3300/0 conn(0x7f07800d3900 0x7f07800b6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 --2- 172.21.15.124:0/1020133334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f078015d330 0x7f07800b8b80 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.214 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 -- 172.21.15.124:0/1020133334 >> 172.21.15.124:0/1020133334 conn(0x7f0780072020 msgr2=0x7f0780072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 -- 172.21.15.124:0/1020133334 shutdown_connections 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.210+0000 7f0787017640 1 -- 172.21.15.124:0/1020133334 wait complete. 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.211+0000 7f0787017640 1 Processor -- start 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.211+0000 7f0787017640 1 -- start start 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07800d2910 0x7f07800b1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.215 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 0x7f078014daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078014c1f0 con 0x7f07800d3900 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078014ca50 con 0x7f078015d330 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0787017640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078014cd30 con 0x7f07800d2910 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f077ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f077ffff640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41516/0 (socket says 172.21.15.124:41516) 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f077ffff640 1 -- 172.21.15.124:0/1478551943 learned_addr learned my addr 172.21.15.124:0/1478551943 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.216 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f07855b4640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 0x7f078014daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.212+0000 7f0784db3640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07800d2910 0x7f07800b1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 -- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07800d2910 msgr2=0x7f07800b1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07800d2910 0x7f07800b1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 -- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 msgr2=0x7f078014daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 0x7f078014daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 -- 172.21.15.124:0/1478551943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774009400 con 0x7f07800d3900 2023-12-13T23:57:55.217 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f07855b4640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 0x7f078014daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077ffff640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c430 tx=0x7f076c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0074e0 con 0x7f07800d3900 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c007660 con 0x7f07800d3900 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c004a20 con 0x7f07800d3900 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f078014d010 con 0x7f07800d3900 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.213+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f078014d5d0 con 0x7f07800d3900 2023-12-13T23:57:55.218 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.214+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f078014d7f0 con 0x7f07800d3900 2023-12-13T23:57:55.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.215+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f076c004b80 con 0x7f07800d3900 2023-12-13T23:57:55.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.215+0000 7f077d7fa640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 0x7f076406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.219 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.215+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f076c089520 con 0x7f07800d3900 2023-12-13T23:57:55.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.215+0000 7f0784db3640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 0x7f076406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.216+0000 7f0784db3640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 0x7f076406d680 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0774036000 tx=0x7f077403d040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.220 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.219+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f076c0545e0 con 0x7f07800d3900 2023-12-13T23:57:55.384 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.384+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f07800d23f0 con 0x7f076406b1c0 2023-12-13T23:57:55.385 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.385+0000 7f077d7fa640 1 -- 172.21.15.124:0/1478551943 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0774031b30 con 0x7f076406b1c0 2023-12-13T23:57:55.385 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 4 to deep-scrub 2023-12-13T23:57:55.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 msgr2=0x7f076406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 0x7f076406d680 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f0774036000 tx=0x7f077403d040 comp rx=0 tx=0).stop 2023-12-13T23:57:55.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 msgr2=0x7f078014eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f076c00c430 tx=0x7f076c00c8f0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 shutdown_connections 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f076406b1c0 0x7f076406d680 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f078015d330 0x7f078014daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07800d3900 0x7f078014eef0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 --2- 172.21.15.124:0/1478551943 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f07800d2910 0x7f07800b1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 >> 172.21.15.124:0/1478551943 conn(0x7f0780072020 msgr2=0x7f07800b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 shutdown_connections 2023-12-13T23:57:55.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.387+0000 7f0787017640 1 -- 172.21.15.124:0/1478551943 wait complete. 2023-12-13T23:57:55.397 INFO:tasks.ceph:Scrubbing osd.5 2023-12-13T23:57:55.398 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:55.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.501+0000 7f0b9d1a9640 1 Processor -- start 2023-12-13T23:57:55.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.501+0000 7f0b9d1a9640 1 -- start start 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b9d1a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b9806d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b9d1a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980adae0 con 0x7f0b9815d1d0 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b9d1a9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b980addc0 con 0x7f0b980ad640 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b9d1a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b980ae0a0 con 0x7f0b9806bbf0 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b96d9d640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0b9815d1d0 0x7f0b9806b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41066/0 (socket says 172.21.15.124:41066) 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b96d9d640 1 -- 172.21.15.124:0/2339944173 learned_addr learned my addr 172.21.15.124:0/2339944173 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.502+0000 7f0b9759e640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b9806d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225035742 0 0) 0x7f0b980adae0 con 0x7f0b9815d1d0 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b84003520 con 0x7f0b9815d1d0 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058970731 0 0) 0x7f0b980addc0 con 0x7f0b980ad640 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b980adae0 con 0x7f0b980ad640 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1525125693 0 0) 0x7f0b84003520 con 0x7f0b9815d1d0 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 msgr2=0x7f0b9806d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b9806d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 >> v1:172.21.15.203:6789/0 conn(0x7f0b980ad640 legacy=0x7f0b9806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b980ae380 con 0x7f0b9815d1d0 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f0b880019f0 con 0x7f0b9815d1d0 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 >> v1:172.21.15.124:6789/0 conn(0x7f0b9815d1d0 legacy=0x7f0b9806b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b84003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b84004d00 0x7f0b84005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b980ae0a0 con 0x7f0b84003b30 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8400ba30 con 0x7f0b84004d00 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b8400bbf0 con 0x7f0b9806bbf0 2023-12-13T23:57:55.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b84003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b9759e640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b84004d00 0x7f0b84005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b9659c640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 -- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 msgr2=0x7f0b840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b9806bbf0 0x7f0b840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 -- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b84004d00 msgr2=0x7f0b84005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b84004d00 0x7f0b84005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b96d9d640 1 -- 172.21.15.124:0/2339944173 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b84003520 con 0x7f0b84003b30 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.503+0000 7f0b9759e640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b84004d00 0x7f0b84005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:55.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.504+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b84003f30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800a7c0 tx=0x7f0b840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3c9aab76ad684 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.504+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88034630 con 0x7f0b84003b30 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.504+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b8802f070 con 0x7f0b84003b30 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.504+0000 7f0b95d9b640 1 -- 172.21.15.124:0/2339944173 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b880349f0 con 0x7f0b84003b30 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/2339944173 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 msgr2=0x7f0b84003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 --2- 172.21.15.124:0/2339944173 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b84003f30 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800a7c0 tx=0x7f0b840057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9759e640 1 -- 172.21.15.124:0/2339944173 reap_dead start 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/2339944173 shutdown_connections 2023-12-13T23:57:55.509 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/2339944173 >> 172.21.15.124:0/2339944173 conn(0x7f0b98070e70 msgr2=0x7f0b98071250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/2339944173 shutdown_connections 2023-12-13T23:57:55.510 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.505+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/2339944173 wait complete. 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.506+0000 7f0b9d1a9640 1 Processor -- start 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.506+0000 7f0b9d1a9640 1 -- start start 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b98153cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b98150ab0 0x7f0b98155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b98143ff0 0x7f0b98144470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.514 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b981449b0 con 0x7f0b84003b30 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b981545c0 con 0x7f0b98143ff0 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9d1a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b981548a0 con 0x7f0b98150ab0 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b98153cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b98153cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41536/0 (socket says 172.21.15.124:41536) 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 -- 172.21.15.124:0/1882138188 learned_addr learned my addr 172.21.15.124:0/1882138188 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9759e640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b98143ff0 0x7f0b98144470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9659c640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b98150ab0 0x7f0b98155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 -- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b98150ab0 msgr2=0x7f0b98155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.515 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b98150ab0 0x7f0b98155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 -- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b98143ff0 msgr2=0x7f0b98144470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b98143ff0 0x7f0b98144470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b98154b80 con 0x7f0b84003b30 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9759e640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f0b98143ff0 0x7f0b98144470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b9659c640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f0b98150ab0 0x7f0b98155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.507+0000 7f0b96d9d640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b98153cc0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800a7c0 tx=0x7f0b88001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b880346c0 con 0x7f0b84003b30 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b88034820 con 0x7f0b84003b30 2023-12-13T23:57:55.516 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b98154e60 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b88034b90 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b98154200 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.508+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b980b4460 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.509+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f0b88031030 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.509+0000 7f0b8f7fe640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0b7806b1f0 0x7f0b7806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.509+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0b880ac420 con 0x7f0b84003b30 2023-12-13T23:57:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b8f7fe640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f0b78071050 0x7f0b78073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b78073b20 con 0x7f0b78071050 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b880ac870 con 0x7f0b84003b30 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b9659c640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0b7806b1f0 0x7f0b7806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b9759e640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f0b78071050 0x7f0b78073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b9659c640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0b7806b1f0 0x7f0b7806d6b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b9806e9d0 tx=0x7f0b80009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.510+0000 7f0b9759e640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f0b78071050 0x7f0b78073470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.518 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.511+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== osd.5 v2:172.21.15.203:6824/1613580500 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0b78073b20 con 0x7f0b78071050 2023-12-13T23:57:55.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.524+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 --> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0b980b47e0 con 0x7f0b78071050 2023-12-13T23:57:55.523 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.524+0000 7f0b8f7fe640 1 -- 172.21.15.124:0/1882138188 <== osd.5 v2:172.21.15.203:6824/1613580500 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0b980b47e0 con 0x7f0b78071050 2023-12-13T23:57:55.524 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:55.524 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:55.524 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f0b78071050 msgr2=0x7f0b78073470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6824/1613580500,v1:172.21.15.203:6825/1613580500] conn(0x7f0b78071050 0x7f0b78073470 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0b7806b1f0 msgr2=0x7f0b7806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f0b7806b1f0 0x7f0b7806d6b0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0b9806e9d0 tx=0x7f0b80009040 comp rx=0 tx=0).stop 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 msgr2=0x7f0b98153cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 --2- 172.21.15.124:0/1882138188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b84003b30 0x7f0b98153cc0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f0b8800a7c0 tx=0x7f0b88001d00 comp rx=0 tx=0).stop 2023-12-13T23:57:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9759e640 1 -- 172.21.15.124:0/1882138188 reap_dead start 2023-12-13T23:57:55.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 shutdown_connections 2023-12-13T23:57:55.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 >> 172.21.15.124:0/1882138188 conn(0x7f0b98070e70 msgr2=0x7f0b980b0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 shutdown_connections 2023-12-13T23:57:55.527 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.525+0000 7f0b9d1a9640 1 -- 172.21.15.124:0/1882138188 wait complete. 2023-12-13T23:57:55.535 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-13T23:57:55.634 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.635+0000 7f7de095a640 1 Processor -- start 2023-12-13T23:57:55.635 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.635+0000 7f7de095a640 1 -- start start 2023-12-13T23:57:55.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7de095a640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ddc143490 0x7f7ddc0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7de095a640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f7ddc15abf0 0x7f7ddc0b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7de095a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ddc15e2a0 con 0x7f7ddc15e730 2023-12-13T23:57:55.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7de095a640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ddc15b520 con 0x7f7ddc143490 2023-12-13T23:57:55.637 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7de095a640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ddc143870 con 0x7f7ddc15abf0 2023-12-13T23:57:55.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ddc143490 0x7f7ddc0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7ddc143490 0x7f7ddc0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41540/0 (socket says 172.21.15.124:41540) 2023-12-13T23:57:55.638 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 -- 172.21.15.124:0/4088225055 learned_addr learned my addr 172.21.15.124:0/4088225055 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7ddad9d640 1 --2- 172.21.15.124:0/4088225055 >> v2:172.21.15.203:3300/0 conn(0x7f7ddc15abf0 0x7f7ddc0b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 -- 172.21.15.124:0/4088225055 >> v2:172.21.15.203:3300/0 conn(0x7f7ddc15abf0 msgr2=0x7f7ddc0b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 --2- 172.21.15.124:0/4088225055 >> v2:172.21.15.203:3300/0 conn(0x7f7ddc15abf0 0x7f7ddc0b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 -- 172.21.15.124:0/4088225055 >> v1:172.21.15.203:6789/0 conn(0x7f7ddc15e730 legacy=0x7f7ddc15a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.641 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.636+0000 7f7dda59c640 1 -- 172.21.15.124:0/4088225055 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc15b240 con 0x7f7ddc143490 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7dda59c640 1 --2- 172.21.15.124:0/4088225055 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc143490 0x7f7ddc0b0cf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0b4ca0 tx=0x7f7dcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cbdb0f0008c3ed server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7dd9d9b640 1 -- 172.21.15.124:0/4088225055 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc031870 con 0x7f7ddc143490 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7dd9d9b640 1 -- 172.21.15.124:0/4088225055 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc0319d0 con 0x7f7ddc143490 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7dd9d9b640 1 -- 172.21.15.124:0/4088225055 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc031d60 con 0x7f7ddc143490 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 -- 172.21.15.124:0/4088225055 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc143490 msgr2=0x7f7ddc0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 --2- 172.21.15.124:0/4088225055 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc143490 0x7f7ddc0b0cf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0b4ca0 tx=0x7f7dcc02b910 comp rx=0 tx=0).stop 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 -- 172.21.15.124:0/4088225055 shutdown_connections 2023-12-13T23:57:55.642 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 --2- 172.21.15.124:0/4088225055 >> v2:172.21.15.203:3300/0 conn(0x7f7ddc15abf0 0x7f7ddc0b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 --2- 172.21.15.124:0/4088225055 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc143490 0x7f7ddc0b0cf0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.637+0000 7f7de095a640 1 -- 172.21.15.124:0/4088225055 >> 172.21.15.124:0/4088225055 conn(0x7f7ddc06a4d0 msgr2=0x7f7ddc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.638+0000 7f7de095a640 1 -- 172.21.15.124:0/4088225055 shutdown_connections 2023-12-13T23:57:55.643 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.638+0000 7f7de095a640 1 -- 172.21.15.124:0/4088225055 wait complete. 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 Processor -- start 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 -- start start 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ddc143490 0x7f7ddc140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 0x7f7ddc159520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.646 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc140610 con 0x7f7ddc15e730 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc13cf40 con 0x7f7ddc15abf0 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.639+0000 7f7de095a640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ddc13c180 con 0x7f7ddc143490 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41552/0 (socket says 172.21.15.124:41552) 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 -- 172.21.15.124:0/1861140647 learned_addr learned my addr 172.21.15.124:0/1861140647 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dd3fff640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 0x7f7ddc159520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dda59c640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ddc143490 0x7f7ddc140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 -- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ddc143490 msgr2=0x7f7ddc140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.647 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ddc143490 0x7f7ddc140030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 -- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 msgr2=0x7f7ddc159520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 0x7f7ddc159520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 -- 172.21.15.124:0/1861140647 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0094d0 con 0x7f7ddc15e730 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dd3fff640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 0x7f7ddc159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7ddad9d640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400cd70 tx=0x7f7dd400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4003b90 con 0x7f7ddc15e730 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd4003cf0 con 0x7f7ddc15e730 2023-12-13T23:57:55.648 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ddc13c460 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd4004240 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.640+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ddc13ca20 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.641+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ddc13cc40 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.642+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f7dd40043a0 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.642+0000 7f7dd2ffd640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 0x7f7dbc06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.642+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7dd4012070 con 0x7f7ddc15e730 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.642+0000 7f7dda59c640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 0x7f7dbc06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.643+0000 7f7dda59c640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 0x7f7dbc06d680 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036000 tx=0x7f7dcc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.649 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.646+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f7dd4055140 con 0x7f7ddc15e730 2023-12-13T23:57:55.812 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.813+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ddc13c2f0 con 0x7f7dbc06b1c0 2023-12-13T23:57:55.813 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.813+0000 7f7dd2ffd640 1 -- 172.21.15.124:0/1861140647 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7ddc13c2f0 con 0x7f7dbc06b1c0 2023-12-13T23:57:55.814 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 5 to deep-scrub 2023-12-13T23:57:55.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 msgr2=0x7f7dbc06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 0x7f7dbc06d680 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc036000 tx=0x7f7dcc03c040 comp rx=0 tx=0).stop 2023-12-13T23:57:55.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 msgr2=0x7f7ddc158f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7dd400cd70 tx=0x7f7dd400a660 comp rx=0 tx=0).stop 2023-12-13T23:57:55.815 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 shutdown_connections 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f7dbc06b1c0 0x7f7dbc06d680 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f7ddc15abf0 0x7f7ddc159520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7ddc15e730 0x7f7ddc158f50 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 --2- 172.21.15.124:0/1861140647 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f7ddc143490 0x7f7ddc140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 >> 172.21.15.124:0/1861140647 conn(0x7f7ddc06a4d0 msgr2=0x7f7ddc15bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 shutdown_connections 2023-12-13T23:57:55.816 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.815+0000 7f7de095a640 1 -- 172.21.15.124:0/1861140647 wait complete. 2023-12-13T23:57:55.825 INFO:tasks.ceph:Scrubbing osd.6 2023-12-13T23:57:55.826 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:55.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.929+0000 7fafea21e640 1 Processor -- start 2023-12-13T23:57:55.929 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.929+0000 7fafea21e640 1 -- start start 2023-12-13T23:57:55.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafea21e640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fafe4158ba0 0x7fafe415c160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.932 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafea21e640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe40adbb0 con 0x7fafe4071c80 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafea21e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe40b4520 con 0x7fafe40ac370 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafea21e640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe40aecf0 con 0x7fafe4158ba0 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafe3fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fafe4071c80 0x7fafe40b2200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41068/0 (socket says 172.21.15.124:41068) 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafe3fff640 1 -- 172.21.15.124:0/708634025 learned_addr learned my addr 172.21.15.124:0/708634025 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafe37fe640 1 --2- 172.21.15.124:0/708634025 >> v2:172.21.15.203:3300/0 conn(0x7fafe4158ba0 0x7fafe415c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.930+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693219843 0 0) 0x7fafe40adbb0 con 0x7fafe4071c80 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc003520 con 0x7fafe4071c80 2023-12-13T23:57:55.933 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832306720 0 0) 0x7fafcc003520 con 0x7fafe4071c80 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 >> v2:172.21.15.203:3300/0 conn(0x7fafe4158ba0 msgr2=0x7fafe415c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 --2- 172.21.15.124:0/708634025 >> v2:172.21.15.203:3300/0 conn(0x7fafe4158ba0 0x7fafe415c160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 >> v1:172.21.15.203:6789/0 conn(0x7fafe40ac370 legacy=0x7fafe40ae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe415c6a0 con 0x7fafe4071c80 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fafdc001a10 con 0x7fafe4071c80 2023-12-13T23:57:55.934 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 >> v1:172.21.15.124:6789/0 conn(0x7fafe4071c80 legacy=0x7fafe40b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc0047c0 0x7fafcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafcc005180 0x7fafcc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40aecf0 con 0x7fafcc003970 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc00c120 con 0x7fafcc0047c0 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafcc00c2e0 con 0x7fafcc005180 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc0047c0 0x7fafcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.935 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe3fff640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafcc005180 0x7fafcc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 -- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafcc005180 msgr2=0x7fafcc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafcc005180 0x7fafcc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 -- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc0047c0 msgr2=0x7fafcc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc0047c0 0x7fafcc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe3fff640 1 -- 172.21.15.124:0/708634025 reap_dead start 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe37fe640 1 -- 172.21.15.124:0/708634025 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafcc003520 con 0x7fafcc003970 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.931+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafcc0047c0 0x7fafcc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.932+0000 7fafe37fe640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafcc003d90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009ac0 tx=0x7fafcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c75e3d18db22c9 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.936 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.932+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd402c6d0 con 0x7fafcc003970 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.932+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd402c830 con 0x7fafcc003970 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.932+0000 7fafe27fc640 1 -- 172.21.15.124:0/708634025 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd402cbc0 con 0x7fafcc003970 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 -- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 msgr2=0x7fafcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafcc003d90 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fafd4009ac0 tx=0x7fafcc0057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 -- 172.21.15.124:0/708634025 shutdown_connections 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 --2- 172.21.15.124:0/708634025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafcc003d90 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 -- 172.21.15.124:0/708634025 >> 172.21.15.124:0/708634025 conn(0x7fafe40b4720 msgr2=0x7fafe40b4b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 -- 172.21.15.124:0/708634025 shutdown_connections 2023-12-13T23:57:55.937 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.933+0000 7fafea21e640 1 -- 172.21.15.124:0/708634025 wait complete. 2023-12-13T23:57:55.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.934+0000 7fafea21e640 1 Processor -- start 2023-12-13T23:57:55.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.934+0000 7fafea21e640 1 -- start start 2023-12-13T23:57:55.946 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafe4151190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafe41f5580 0x7fafe4150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafe4153970 0x7fafe41f4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe41542e0 con 0x7fafcc003970 2023-12-13T23:57:55.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40aff20 con 0x7fafe4153970 2023-12-13T23:57:55.947 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe40b0200 con 0x7fafe41f5580 2023-12-13T23:57:55.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafe4151190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafe4151190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41570/0 (socket says 172.21.15.124:41570) 2023-12-13T23:57:55.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 -- 172.21.15.124:0/2428957681 learned_addr learned my addr 172.21.15.124:0/2428957681 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:55.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafe41f5580 0x7fafe4150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.948 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe3fff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafe4153970 0x7fafe41f4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 -- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafe41f5580 msgr2=0x7fafe4150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafe41f5580 0x7fafe4150790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 -- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafe4153970 msgr2=0x7fafe41f4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.949 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafe4153970 0x7fafe41f4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe40b04e0 con 0x7fafcc003970 2023-12-13T23:57:55.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fafe41f5580 0x7fafe4150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:55.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe3fff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fafe4153970 0x7fafe41f4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:55.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafe37fe640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafe4151190 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fafd402c270 tx=0x7fafd4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.950 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003810 con 0x7fafcc003970 2023-12-13T23:57:55.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd4003970 con 0x7fafcc003970 2023-12-13T23:57:55.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe40b07c0 con 0x7fafcc003970 2023-12-13T23:57:55.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4003ce0 con 0x7fafcc003970 2023-12-13T23:57:55.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.935+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe4150cd0 con 0x7fafcc003970 2023-12-13T23:57:55.951 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.936+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fafe40adde0 con 0x7fafcc003970 2023-12-13T23:57:55.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fafd4040660 con 0x7fafcc003970 2023-12-13T23:57:55.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd006b1c0 0x7fafd006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fafd40abfc0 con 0x7fafcc003970 2023-12-13T23:57:55.952 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7fafd0071020 0x7fafd0073440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:55.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fafd0073af0 con 0x7fafd0071020 2023-12-13T23:57:55.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafd40ac410 con 0x7fafcc003970 2023-12-13T23:57:55.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.937+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd006b1c0 0x7fafd006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.953 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.938+0000 7fafe3fff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7fafd0071020 0x7fafd0073440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:55.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.938+0000 7fafe2ffd640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd006b1c0 0x7fafd006d680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fafe4159ef0 tx=0x7fafd8009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.938+0000 7fafe3fff640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7fafd0071020 0x7fafd0073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:55.954 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.939+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== osd.6 v2:172.21.15.203:6808/3358010181 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fafd0073af0 con 0x7fafd0071020 2023-12-13T23:57:55.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.951+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 --> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fafe40b4240 con 0x7fafd0071020 2023-12-13T23:57:55.955 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafcbfff640 1 -- 172.21.15.124:0/2428957681 <== osd.6 v2:172.21.15.203:6808/3358010181 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fafdc00d040 con 0x7fafd0071020 2023-12-13T23:57:55.956 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:55.956 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:55.956 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7fafd0071020 msgr2=0x7fafd0073440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6808/3358010181,v1:172.21.15.203:6810/3358010181] conn(0x7fafd0071020 0x7fafd0073440 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd006b1c0 msgr2=0x7fafd006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fafd006b1c0 0x7fafd006d680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fafe4159ef0 tx=0x7fafd8009040 comp rx=0 tx=0).stop 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 msgr2=0x7fafe4151190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 --2- 172.21.15.124:0/2428957681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafcc003970 0x7fafe4151190 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fafd402c270 tx=0x7fafd4004730 comp rx=0 tx=0).stop 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafe3fff640 1 -- 172.21.15.124:0/2428957681 reap_dead start 2023-12-13T23:57:55.958 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 shutdown_connections 2023-12-13T23:57:55.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.952+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 >> 172.21.15.124:0/2428957681 conn(0x7fafe40b4720 msgr2=0x7fafe41571a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:55.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.953+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 shutdown_connections 2023-12-13T23:57:55.959 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:55.953+0000 7fafea21e640 1 -- 172.21.15.124:0/2428957681 wait complete. 2023-12-13T23:57:55.962 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-13T23:57:56.063 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.064+0000 7f3c8e890640 1 Processor -- start 2023-12-13T23:57:56.064 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.064+0000 7f3c8e890640 1 -- start start 2023-12-13T23:57:56.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c8e890640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c880b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c8e890640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8815e2a0 con 0x7f3c8815e730 2023-12-13T23:57:56.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c8e890640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8815b240 con 0x7f3c8815abf0 2023-12-13T23:57:56.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c8e890640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8815b520 con 0x7f3c88143490 2023-12-13T23:57:56.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c877fe640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3c8815e730 0x7f3c8815a4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:41082/0 (socket says 172.21.15.124:41082) 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c877fe640 1 -- 172.21.15.124:0/1753118014 learned_addr learned my addr 172.21.15.124:0/1753118014 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 229335632 0 0) 0x7f3c8815e2a0 con 0x7f3c8815e730 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c70003520 con 0x7f3c8815e730 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c87fff640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c880b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312363543 0 0) 0x7f3c70003520 con 0x7f3c8815e730 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 msgr2=0x7f3c880b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c880b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.068 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 >> v1:172.21.15.203:6789/0 conn(0x7f3c8815abf0 legacy=0x7f3c880b2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8815dc50 con 0x7f3c8815e730 2023-12-13T23:57:56.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7f3c7c001670 con 0x7f3c8815e730 2023-12-13T23:57:56.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 >> v1:172.21.15.124:6789/0 conn(0x7f3c8815e730 legacy=0x7f3c8815a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c700039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.069 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c70003f20 0x7f3c700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70004d00 0x7f3c70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8815b520 con 0x7f3c70003f20 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c700099a0 con 0x7f3c70004d00 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70009b60 con 0x7f3c88143490 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c70003f20 0x7f3c700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c87fff640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c700039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.070 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.065+0000 7f3c8ce2d640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70004d00 0x7f3c70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 -- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 msgr2=0x7f3c700039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88143490 0x7f3c700039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 -- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70004d00 msgr2=0x7f3c70005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70004d00 0x7f3c70005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 -- 172.21.15.124:0/1753118014 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c70003520 con 0x7f3c70003f20 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c70003f20 0x7f3c700047c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001050 tx=0x7f3c700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915d1fd82c391d16 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.066+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c005ad0 con 0x7f3c70003f20 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c005c30 con 0x7f3c70003f20 2023-12-13T23:57:56.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c86ffd640 1 -- 172.21.15.124:0/1753118014 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c02cad0 con 0x7f3c70003f20 2023-12-13T23:57:56.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 -- 172.21.15.124:0/1753118014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c70003f20 msgr2=0x7f3c700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1753118014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c70003f20 0x7f3c700047c0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001050 tx=0x7f3c700057f0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8ce2d640 1 -- 172.21.15.124:0/1753118014 reap_dead start 2023-12-13T23:57:56.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 -- 172.21.15.124:0/1753118014 shutdown_connections 2023-12-13T23:57:56.072 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 -- 172.21.15.124:0/1753118014 >> 172.21.15.124:0/1753118014 conn(0x7f3c8806a4d0 msgr2=0x7f3c8806a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 -- 172.21.15.124:0/1753118014 shutdown_connections 2023-12-13T23:57:56.073 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.067+0000 7f3c8e890640 1 -- 172.21.15.124:0/1753118014 wait complete. 2023-12-13T23:57:56.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.068+0000 7f3c8e890640 1 Processor -- start 2023-12-13T23:57:56.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.068+0000 7f3c8e890640 1 -- start start 2023-12-13T23:57:56.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70003f20 0x7f3c88159830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.076 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88134b30 0x7f3c88134fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c881429d0 con 0x7f3c88142010 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c88159d70 con 0x7f3c70003f20 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c8e890640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c881358f0 con 0x7f3c88134b30 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41598/0 (socket says 172.21.15.124:41598) 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 -- 172.21.15.124:0/1127750008 learned_addr learned my addr 172.21.15.124:0/1127750008 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 -- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88134b30 msgr2=0x7f3c88134fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.077 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c87fff640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70003f20 0x7f3c88159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88134b30 0x7f3c88134fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 -- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70003f20 msgr2=0x7f3c88159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70003f20 0x7f3c88159830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.069+0000 7f3c877fe640 1 -- 172.21.15.124:0/1127750008 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88135bd0 con 0x7f3c88142010 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c877fe640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001050 tx=0x7f3c7c000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c02ccc0 con 0x7f3c88142010 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c7c02ce40 con 0x7f3c88142010 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c88135eb0 con 0x7f3c88142010 2023-12-13T23:57:56.078 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c7c0072b0 con 0x7f3c88142010 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.070+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c88136470 con 0x7f3c88142010 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.071+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c880b4380 con 0x7f3c88142010 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.071+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f3c7c032070 con 0x7f3c88142010 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.071+0000 7f3c84ff9640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 0x7f3c7406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.072+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3c7c0aab90 con 0x7f3c88142010 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.072+0000 7f3c87fff640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 0x7f3c7406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.072+0000 7f3c87fff640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 0x7f3c7406d680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3c780099e0 tx=0x7f3c78008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.079 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.076+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f3c7c075bc0 con 0x7f3c88142010 2023-12-13T23:57:56.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.244+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c8815a690 con 0x7f3c7406b1c0 2023-12-13T23:57:56.244 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.245+0000 7f3c84ff9640 1 -- 172.21.15.124:0/1127750008 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3c8815a690 con 0x7f3c7406b1c0 2023-12-13T23:57:56.245 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 6 to deep-scrub 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 msgr2=0x7f3c7406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 0x7f3c7406d680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3c780099e0 tx=0x7f3c78008040 comp rx=0 tx=0).stop 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 msgr2=0x7f3c88142490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f3c7c001050 tx=0x7f3c7c000fe0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 shutdown_connections 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f3c7406b1c0 0x7f3c7406d680 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f3c88134b30 0x7f3c88134fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3c88142010 0x7f3c88142490 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 --2- 172.21.15.124:0/1127750008 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f3c70003f20 0x7f3c88159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 >> 172.21.15.124:0/1127750008 conn(0x7f3c8806a4d0 msgr2=0x7f3c8815c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 shutdown_connections 2023-12-13T23:57:56.248 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.247+0000 7f3c8e890640 1 -- 172.21.15.124:0/1127750008 wait complete. 2023-12-13T23:57:56.257 INFO:tasks.ceph:Scrubbing osd.7 2023-12-13T23:57:56.257 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-13T23:57:56.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.362+0000 7f9b1c316640 1 Processor -- start 2023-12-13T23:57:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.362+0000 7f9b1c316640 1 -- start start 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1c316640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9b14143490 0x7f9b140b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1c316640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 0x7f9b140b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1c316640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b1415e2a0 con 0x7f9b1415a4e0 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1c316640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b1415b520 con 0x7f9b14143490 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1c316640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b14143870 con 0x7f9b1415e730 2023-12-13T23:57:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9b14143490 0x7f9b140b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9b14143490 0x7f9b140b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41612/0 (socket says 172.21.15.124:41612) 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 -- 172.21.15.124:0/3903871567 learned_addr learned my addr 172.21.15.124:0/3903871567 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b198b1640 1 --2- 172.21.15.124:0/3903871567 >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 0x7f9b140b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 -- 172.21.15.124:0/3903871567 >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 msgr2=0x7f9b140b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/3903871567 >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 0x7f9b140b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 -- 172.21.15.124:0/3903871567 >> v1:172.21.15.203:6789/0 conn(0x7f9b1415a4e0 legacy=0x7f9b1415a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b1a0b2640 1 -- 172.21.15.124:0/3903871567 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1415b240 con 0x7f9b14143490 2023-12-13T23:57:56.365 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.363+0000 7f9b198b1640 1 --2- 172.21.15.124:0/3903871567 >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 0x7f9b140b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/3903871567 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b14143490 0x7f9b140b0cf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9b140aeca0 tx=0x7f9b1002e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7a0631ca7538a5 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b190b0640 1 -- 172.21.15.124:0/3903871567 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b1002ee90 con 0x7f9b14143490 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b190b0640 1 -- 172.21.15.124:0/3903871567 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b10036440 con 0x7f9b14143490 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b190b0640 1 -- 172.21.15.124:0/3903871567 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b100367d0 con 0x7f9b14143490 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 -- 172.21.15.124:0/3903871567 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b14143490 msgr2=0x7f9b140b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 --2- 172.21.15.124:0/3903871567 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b14143490 0x7f9b140b0cf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f9b140aeca0 tx=0x7f9b1002e0e0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 -- 172.21.15.124:0/3903871567 shutdown_connections 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 --2- 172.21.15.124:0/3903871567 >> v2:172.21.15.203:3300/0 conn(0x7f9b1415e730 0x7f9b140b1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.366 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 --2- 172.21.15.124:0/3903871567 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b14143490 0x7f9b140b0cf0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.364+0000 7f9b1c316640 1 -- 172.21.15.124:0/3903871567 >> 172.21.15.124:0/3903871567 conn(0x7f9b1406a4d0 msgr2=0x7f9b1406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.365+0000 7f9b1c316640 1 -- 172.21.15.124:0/3903871567 shutdown_connections 2023-12-13T23:57:56.367 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.365+0000 7f9b1c316640 1 -- 172.21.15.124:0/3903871567 wait complete. 2023-12-13T23:57:56.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 Processor -- start 2023-12-13T23:57:56.370 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 -- start start 2023-12-13T23:57:56.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b14143490 0x7f9b140adce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 0x7f9b1413baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b1415e730 0x7f9b14140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.371 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1413f270 con 0x7f9b1415a4e0 2023-12-13T23:57:56.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1413cb70 con 0x7f9b14143490 2023-12-13T23:57:56.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b1c316640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b1413f700 con 0x7f9b1415e730 2023-12-13T23:57:56.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b198b1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 0x7f9b1413baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.372 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b198b1640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 0x7f9b1413baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41626/0 (socket says 172.21.15.124:41626) 2023-12-13T23:57:56.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.366+0000 7f9b198b1640 1 -- 172.21.15.124:0/2509641639 learned_addr learned my addr 172.21.15.124:0/2509641639 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b14143490 0x7f9b140adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b1a8b3640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b1415e730 0x7f9b14140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.373 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 -- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b1415e730 msgr2=0x7f9b14140ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f9b1415e730 0x7f9b14140ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 -- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b14143490 msgr2=0x7f9b140adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b14143490 0x7f9b140adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b1000b770 con 0x7f9b1415a4e0 2023-12-13T23:57:56.374 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f9b14143490 0x7f9b140adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:56.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b198b1640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 0x7f9b1413baf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c7f0 tx=0x7f9b0800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08007820 con 0x7f9b1415a4e0 2023-12-13T23:57:56.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b08007980 con 0x7f9b1415a4e0 2023-12-13T23:57:56.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b1413ec40 con 0x7f9b1415a4e0 2023-12-13T23:57:56.375 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b08007cf0 con 0x7f9b1415a4e0 2023-12-13T23:57:56.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.367+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b1413ef00 con 0x7f9b1415a4e0 2023-12-13T23:57:56.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.368+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9b1413eac0 con 0x7f9b1415a4e0 2023-12-13T23:57:56.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.368+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f9b080113f0 con 0x7f9b1415a4e0 2023-12-13T23:57:56.376 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b06ffd640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9af806b1f0 0x7f9af806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f9b08089970 con 0x7f9b1415a4e0 2023-12-13T23:57:56.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b06ffd640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f9af8071050 0x7f9af8073470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9af8073b20 con 0x7f9af8071050 2023-12-13T23:57:56.377 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=53) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b0808e2c0 con 0x7f9b1415a4e0 2023-12-13T23:57:56.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9af806b1f0 0x7f9af806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.369+0000 7f9b1a8b3640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f9af8071050 0x7f9af8073470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.370+0000 7f9b1a0b2640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9af806b1f0 0x7f9af806d6b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9b10038000 tx=0x7f9b10008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.378 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.370+0000 7f9b1a8b3640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f9af8071050 0x7f9af8073470 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.379 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.370+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== osd.7 v2:172.21.15.203:6800/2404558368 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f9af8073b20 con 0x7f9af8071050 2023-12-13T23:57:56.382 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.383+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 --> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f9b1415df30 con 0x7f9af8071050 2023-12-13T23:57:56.383 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b06ffd640 1 -- 172.21.15.124:0/2509641639 <== osd.7 v2:172.21.15.203:6800/2404558368 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f9b0c005650 con 0x7f9af8071050 2023-12-13T23:57:56.383 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2023-12-13T23:57:56.384 INFO:teuthology.orchestra.run.smithi124.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-13T23:57:56.384 INFO:teuthology.orchestra.run.smithi124.stdout:} 2023-12-13T23:57:56.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f9af8071050 msgr2=0x7f9af8073470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6800/2404558368,v1:172.21.15.203:6801/2404558368] conn(0x7f9af8071050 0x7f9af8073470 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.386 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9af806b1f0 msgr2=0x7f9af806d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f9af806b1f0 0x7f9af806d6b0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f9b10038000 tx=0x7f9b10008040 comp rx=0 tx=0).stop 2023-12-13T23:57:56.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 msgr2=0x7f9b1413baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 --2- 172.21.15.124:0/2509641639 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9b1415a4e0 0x7f9b1413baf0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f9b0800c7f0 tx=0x7f9b0800ccb0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.387 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1a8b3640 1 -- 172.21.15.124:0/2509641639 reap_dead start 2023-12-13T23:57:56.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 shutdown_connections 2023-12-13T23:57:56.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 >> 172.21.15.124:0/2509641639 conn(0x7f9b1406a4d0 msgr2=0x7f9b14070290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 shutdown_connections 2023-12-13T23:57:56.388 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.384+0000 7f9b1c316640 1 -- 172.21.15.124:0/2509641639 wait complete. 2023-12-13T23:57:56.394 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-13T23:57:56.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.495+0000 7f2fd444f640 1 Processor -- start 2023-12-13T23:57:56.495 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.495+0000 7f2fd444f640 1 -- start start 2023-12-13T23:57:56.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fcc144980 0x7f2fcc15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 0x7f2fcc0cc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc0b4370 0x7f2fcc0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc071770 con 0x7f2fcc144980 2023-12-13T23:57:56.498 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc0b4ce0 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.496+0000 7f2fd444f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0ccdc0 con 0x7f2fcc0b4370 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fcc144980 0x7f2fcc15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2fcc144980 0x7f2fcc15c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41640/0 (socket says 172.21.15.124:41640) 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 -- 172.21.15.124:0/1428203614 learned_addr learned my addr 172.21.15.124:0/1428203614 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd19ea640 1 --2- 172.21.15.124:0/1428203614 >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 0x7f2fcc0cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd29ec640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc0b4370 0x7f2fcc0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 -- 172.21.15.124:0/1428203614 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc0b4370 msgr2=0x7f2fcc0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.499 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc0b4370 0x7f2fcc0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 -- 172.21.15.124:0/1428203614 >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 msgr2=0x7f2fcc0cc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/1428203614 >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 0x7f2fcc0cc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd21eb640 1 -- 172.21.15.124:0/1428203614 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0cd0a0 con 0x7f2fcc144980 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.497+0000 7f2fd19ea640 1 --2- 172.21.15.124:0/1428203614 >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 0x7f2fcc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc144980 0x7f2fcc15c7c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc15b600 tx=0x7f2fbc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5563ab36b26280 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd11e9640 1 -- 172.21.15.124:0/1428203614 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc031870 con 0x7f2fcc144980 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd11e9640 1 -- 172.21.15.124:0/1428203614 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fbc0319d0 con 0x7f2fcc144980 2023-12-13T23:57:56.500 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd11e9640 1 -- 172.21.15.124:0/1428203614 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc031d60 con 0x7f2fcc144980 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 -- 172.21.15.124:0/1428203614 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc144980 msgr2=0x7f2fcc15c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc144980 0x7f2fcc15c7c0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc15b600 tx=0x7f2fbc02b910 comp rx=0 tx=0).stop 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 -- 172.21.15.124:0/1428203614 shutdown_connections 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc0b4370 0x7f2fcc0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 --2- 172.21.15.124:0/1428203614 >> v2:172.21.15.203:3300/0 conn(0x7f2fcc0ceb70 0x7f2fcc0cc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 --2- 172.21.15.124:0/1428203614 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc144980 0x7f2fcc15c7c0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 -- 172.21.15.124:0/1428203614 >> 172.21.15.124:0/1428203614 conn(0x7f2fcc06a4d0 msgr2=0x7f2fcc06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.501 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 -- 172.21.15.124:0/1428203614 shutdown_connections 2023-12-13T23:57:56.502 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.498+0000 7f2fd444f640 1 -- 172.21.15.124:0/1428203614 wait complete. 2023-12-13T23:57:56.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 Processor -- start 2023-12-13T23:57:56.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 -- start start 2023-12-13T23:57:56.504 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 0x7f2fcc157810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 0x7f2fcc13e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0af870 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0afb50 con 0x7f2fcc0b4370 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd444f640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc0afe30 con 0x7f2fcc144980 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd19ea640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd19ea640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41650/0 (socket says 172.21.15.124:41650) 2023-12-13T23:57:56.505 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.500+0000 7f2fd19ea640 1 -- 172.21.15.124:0/726077878 learned_addr learned my addr 172.21.15.124:0/726077878 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd29ec640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 0x7f2fcc13e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 0x7f2fcc157810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 -- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 msgr2=0x7f2fcc13e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 0x7f2fcc13e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 -- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 msgr2=0x7f2fcc157810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 0x7f2fcc157810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 -- 172.21.15.124:0/726077878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc0094d0 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 0x7f2fcc157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:56.506 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd29ec640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 0x7f2fcc13e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd19ea640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800e730 tx=0x7f2fc800ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8004230 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc8004390 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc0b0110 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc800c8f0 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.501+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc157d50 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.502+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc069400 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.507 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.502+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f2fc800ca50 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.503+0000 7f2fc2ffd640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 0x7f2fb006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.503+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f2fc8089240 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.503+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 0x7f2fb006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.503+0000 7f2fd21eb640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 0x7f2fb006d680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc03c040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.508 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.507+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f2fc8054300 con 0x7f2fcc0ceb70 2023-12-13T23:57:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.674+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f2fcc144d60 con 0x7f2fb006b1c0 2023-12-13T23:57:56.674 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.675+0000 7f2fc2ffd640 1 -- 172.21.15.124:0/726077878 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2fbc03a420 con 0x7f2fb006b1c0 2023-12-13T23:57:56.674 INFO:teuthology.orchestra.run.smithi124.stderr:instructed osd(s) 7 to deep-scrub 2023-12-13T23:57:56.676 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 msgr2=0x7f2fb006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 0x7f2fb006d680 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc036000 tx=0x7f2fbc03c040 comp rx=0 tx=0).stop 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 msgr2=0x7f2fcc0b1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800e730 tx=0x7f2fc800ebf0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 shutdown_connections 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f2fb006b1c0 0x7f2fb006d680 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f2fcc144980 0x7f2fcc13e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2fcc0ceb70 0x7f2fcc0b1140 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.677 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 --2- 172.21.15.124:0/726077878 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f2fcc0b4370 0x7f2fcc157810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 >> 172.21.15.124:0/726077878 conn(0x7f2fcc06a4d0 msgr2=0x7f2fcc143c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 shutdown_connections 2023-12-13T23:57:56.678 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.677+0000 7f2fd444f640 1 -- 172.21.15.124:0/726077878 wait complete. 2023-12-13T23:57:56.687 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:57:56.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.788+0000 7f01af1b6640 1 Processor -- start 2023-12-13T23:57:56.788 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.788+0000 7f01af1b6640 1 -- start start 2023-12-13T23:57:56.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01af1b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8155790 0x7f01a806f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01af1b6640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f01a806abe0 0x7f01a815ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.790 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01af1b6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a8155b70 con 0x7f01a815b5e0 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01af1b6640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01a806ee30 con 0x7f01a806abe0 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01af1b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a806f3b0 con 0x7f01a8155790 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f01a806abe0 0x7f01a815ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f01a806abe0 0x7f01a815ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41654/0 (socket says 172.21.15.124:41654) 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 -- 172.21.15.124:0/826677343 learned_addr learned my addr 172.21.15.124:0/826677343 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01acf52640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8155790 0x7f01a806f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 -- 172.21.15.124:0/826677343 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8155790 msgr2=0x7f01a806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.791 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8155790 0x7f01a806f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 -- 172.21.15.124:0/826677343 >> v1:172.21.15.203:6789/0 conn(0x7f01a815b5e0 legacy=0x7f01a806a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.789+0000 7f01ad753640 1 -- 172.21.15.124:0/826677343 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a815c170 con 0x7f01a806abe0 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.790+0000 7f01ad753640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a815ba60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009e50 tx=0x7f01a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66d6cbcda494003 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.790+0000 7f01a77fe640 1 -- 172.21.15.124:0/826677343 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a002ca70 con 0x7f01a806abe0 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.790+0000 7f01a77fe640 1 -- 172.21.15.124:0/826677343 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01a002cbd0 con 0x7f01a806abe0 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.790+0000 7f01a77fe640 1 -- 172.21.15.124:0/826677343 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01a00343f0 con 0x7f01a806abe0 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 -- 172.21.15.124:0/826677343 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 msgr2=0x7f01a815ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a815ba60 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f01a0009e50 tx=0x7f01a002bed0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.792 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 -- 172.21.15.124:0/826677343 shutdown_connections 2023-12-13T23:57:56.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a815ba60 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 --2- 172.21.15.124:0/826677343 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a8155790 0x7f01a806f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 -- 172.21.15.124:0/826677343 >> 172.21.15.124:0/826677343 conn(0x7f01a80b1e50 msgr2=0x7f01a80b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 -- 172.21.15.124:0/826677343 shutdown_connections 2023-12-13T23:57:56.793 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.791+0000 7f01af1b6640 1 -- 172.21.15.124:0/826677343 wait complete. 2023-12-13T23:57:56.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.792+0000 7f01af1b6640 1 Processor -- start 2023-12-13T23:57:56.795 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.792+0000 7f01af1b6640 1 -- start start 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a8155790 0x7f01a81506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815b5e0 0x7f01a815a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8159260 con 0x7f01a806abe0 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8136540 con 0x7f01a8155790 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01af1b6640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01a8137270 con 0x7f01a815b5e0 2023-12-13T23:57:56.796 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:41660/0 (socket says 172.21.15.124:41660) 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 -- 172.21.15.124:0/1187119896 learned_addr learned my addr 172.21.15.124:0/1187119896 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 -- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815b5e0 msgr2=0x7f01a815a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815b5e0 0x7f01a815a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 -- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a8155790 msgr2=0x7f01a81506a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01a7fff640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a8155790 0x7f01a81506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a8155790 0x7f01a81506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 -- 172.21.15.124:0/1187119896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a0009b30 con 0x7f01a806abe0 2023-12-13T23:57:56.797 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01acf52640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d5f0 tx=0x7f019c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c0125f0 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.793+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f019c012750 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.794+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a81594b0 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.794+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f019c012b00 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.794+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a8159a70 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.794+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a80b2760 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.795+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f019c012c60 con 0x7f01a806abe0 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.795+0000 7f01a57fa640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 0x7f018c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:57:56.798 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.795+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f019c089bc0 con 0x7f01a806abe0 2023-12-13T23:57:56.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.795+0000 7f01a7fff640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 0x7f018c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:57:56.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.796+0000 7f01a7fff640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 0x7f018c06d680 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f01a81498c0 tx=0x7f0194009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:57:56.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.799+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f019c054bf0 con 0x7f01a806abe0 2023-12-13T23:57:56.961 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.962+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01a80cfdf0 con 0x7f018c06b1c0 2023-12-13T23:57:56.964 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.965+0000 7f01a57fa640 1 -- 172.21.15.124:0/1187119896 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51191 (secure 0 0 0) 0x7f0194003a30 con 0x7f018c06b1c0 2023-12-13T23:57:56.964 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:57:56.965 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:57:56.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.966+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 msgr2=0x7f018c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.966+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 0x7f018c06d680 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f01a81498c0 tx=0x7f0194009040 comp rx=0 tx=0).stop 2023-12-13T23:57:56.966 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.966+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 msgr2=0x7f01a8136000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.966+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f019c00d5f0 tx=0x7f019c00dab0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 shutdown_connections 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f018c06b1c0 0x7f018c06d680 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f01a8155790 0x7f01a81506a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f01a815b5e0 0x7f01a815a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 --2- 172.21.15.124:0/1187119896 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f01a806abe0 0x7f01a8136000 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 >> 172.21.15.124:0/1187119896 conn(0x7f01a80b1e50 msgr2=0x7f01a80cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 shutdown_connections 2023-12-13T23:57:56.967 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:57:56.967+0000 7f01af1b6640 1 -- 172.21.15.124:0/1187119896 wait complete. 2023-12-13T23:57:56.977 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":134,"stamp":"2023-12-13T23:57:56.149814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":226476,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748244,"statfs":{"total":773094113280,"available":772862201856,"internally_reserved":0,"allocated":6299648,"data_stored":3294552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-58,"num_write_kb":-38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001821"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":198,"reported_epoch":52,"state":"active+clean","last_fresh":"2023-12-13T23:57:48.229026+0000","last_change":"2023-12-13T23:55:02.358235+0000","last_active":"2023-12-13T23:57:48.229026+0000","last_peered":"2023-12-13T23:57:48.229026+0000","last_clean":"2023-12-13T23:57:48.229026+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:48.229026+0000","last_undegraded":"2023-12-13T23:57:48.229026+0000","last_fullsized":"2023-12-13T23:57:48.229026+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-13T23:55:01.135603+0000","last_clean_scrub_stamp":"2023-12-13T23:55:01.135603+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T10:20:23.743442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.52300000000000002,"15min":0.51900000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68300000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.54600000000000004},"min":{"1min":0.434,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.65300000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.44800000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.495,"15min":0.495},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.63500000000000001,"5min":0.754,"15min":0.754},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.46000000000000002,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70499999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.496}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.375,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.75,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55500000000000005,"15min":0.55800000000000005},"min":{"1min":0.41399999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.621,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51900000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.36399999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.66900000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.41699999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69499999999999995,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.59699999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.46200000000000002,"15min":0.46800000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69399999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46500000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.56200000000000006,"15min":0.57499999999999996},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.78000000000000003,"5min":1.194,"15min":1.194},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.57299999999999995,"15min":0.57699999999999996},"min":{"1min":0.40200000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.73899999999999999,"5min":1.002,"15min":1.002},"last":0.58299999999999996}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.61699999999999999,"15min":0.625},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76800000000000002,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.60699999999999998,"15min":0.61799999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.72199999999999998,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.68100000000000005}]}]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55700000000000005},"min":{"1min":0.45000000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.80600000000000005,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.73199999999999998,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.39600000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.58499999999999996,"15min":0.58899999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77200000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.442,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.70899999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.61799999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38500000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.753,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.46800000000000003,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.73499999999999999,"5min":1.006,"15min":1.006},"last":0.495}]},{"osd":3,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.502,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.73899999999999999,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42499999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.52100000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.504,"15min":0.51100000000000001},"min":{"1min":0.33900000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.432},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.54700000000000004,"15min":0.54900000000000004},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.81000000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.46500000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.312,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.75900000000000001,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.54200000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.33200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.79400000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.61599999999999999,"15min":0.625},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.82699999999999996,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.45100000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.56000000000000005,"15min":0.55600000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.751,"5min":1.03,"15min":1.03},"last":0.90200000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.51700000000000002,"15min":0.51000000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.56599999999999995,"15min":0.56299999999999994},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.82799999999999996,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.81599999999999995}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.45600000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.77400000000000002,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55300000000000005,"15min":0.55200000000000005},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76400000000000001,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.871}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.57799999999999996,"15min":0.57499999999999996},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.82699999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.54400000000000004,"15min":0.53300000000000003},"min":{"1min":0.46800000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.88300000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.36299999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.62,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.41199999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73299999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.51100000000000001,"15min":0.50700000000000001},"min":{"1min":0.36599999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.753,"5min":0.755,"15min":0.755},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.67600000000000005,"5min":0.754,"15min":0.754},"last":0.79200000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58499999999999996,"15min":0.58699999999999997},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.74299999999999999,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.627,"5min":0.58899999999999997,"15min":0.58299999999999996},"min":{"1min":0.42599999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.83999999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51500000000000001,"15min":0.50900000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.434},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.51100000000000001,"15min":0.505},"min":{"1min":0.371,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.76700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.53500000000000003,"15min":0.53800000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65400000000000003,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53900000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.70799999999999996,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.73399999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.53100000000000003,"15min":0.52400000000000002},"min":{"1min":0.437,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.68500000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.377,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56899999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.371,"15min":0.371},"max":{"1min":0.73599999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64700000000000002}]},{"osd":5,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53500000000000003,"15min":0.53400000000000003},"min":{"1min":0.30099999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72599999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.627}]},{"osd":6,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68899999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59699999999999998,"15min":0.59899999999999998},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78000000000000003,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.67300000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:57:01 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.61799999999999999,"15min":0.624},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77200000000000002,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.71699999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574887,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.32800000000000001,"15min":0.32600000000000001},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32000000000000001},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.26200000000000001}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.34799999999999998,"15min":0.34200000000000003},"min":{"1min":0.29299999999999998,"5min":0.249,"15min":0.249},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.38300000000000001}]},{"osd":2,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.33000000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.39700000000000002,"15min":0.39300000000000002},"min":{"1min":0.317,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.627,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.41599999999999998}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.57399999999999995,"15min":0.56799999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.41099999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59599999999999997,"15min":0.59299999999999997},"min":{"1min":0.44,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.85999999999999999,"5min":1.038,"15min":1.038},"last":0.70499999999999996}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.88500000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.58399999999999996,"15min":0.57599999999999996},"min":{"1min":0.41999999999999998,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.55900000000000005}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58199999999999996,"15min":0.57599999999999996},"min":{"1min":0.45000000000000001,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.46500000000000002,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.72199999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.31900000000000001},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.25},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.32600000000000001,"15min":0.32100000000000001},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.41099999999999998}]},{"osd":1,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.34200000000000003,"15min":0.33800000000000002},"min":{"1min":0.26700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48799999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.35799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.374}]},{"osd":3,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.32800000000000001,"15min":0.318},"min":{"1min":0.23899999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.377,"15min":0.373},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.33700000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.57299999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.623,"5min":0.58399999999999996,"15min":0.57799999999999996},"min":{"1min":0.47799999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.66800000000000004}]},{"osd":5,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58199999999999996,"15min":0.57999999999999996},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.627},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.46400000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61599999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.44,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.748,"5min":0.871,"15min":0.871},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59199999999999997,"15min":0.58999999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55800000000000005}]},{"osd":7,"last update":"Wed Dec 13 23:57:04 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.441,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.58799999999999997,"15min":0.58599999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79700000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70899999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.39800000000000002,"15min":0.40899999999999997},"min":{"1min":0.21299999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.59799999999999998,"5min":1.488,"15min":1.488},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.376,"15min":0.377},"min":{"1min":0.23499999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.50900000000000001,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.35499999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.39100000000000001,"15min":0.40100000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51100000000000001,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.28100000000000003,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68700000000000006,"5min":1.536,"15min":1.536},"last":0.31}]},{"osd":3,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53200000000000003,"5min":1.512,"15min":1.512},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.39600000000000002,"15min":0.39900000000000002},"min":{"1min":0.28599999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61799999999999999,"5min":1.401,"15min":1.401},"last":0.26200000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.45300000000000001,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.63100000000000001,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.747,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.44400000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56299999999999994,"15min":0.56799999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67000000000000004,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.60399999999999998,"15min":0.60999999999999999},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.40699999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.59099999999999997,"15min":0.59699999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.76200000000000001,"5min":1.292,"15min":1.292},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.58599999999999997,"15min":0.59099999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.70299999999999996,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.48299999999999998}]},{"osd":7,"last update":"Wed Dec 13 23:57:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82799999999999996,"5min":1.46,"15min":1.46},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.625,"15min":0.627},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.84599999999999997,"5min":1.377,"15min":1.377},"last":0.52200000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28224,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343616,"statfs":{"total":96636764160,"available":96607862784,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.40899999999999997,"15min":0.41699999999999998},"min":{"1min":0.26300000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.53700000000000003,"5min":1.137,"15min":1.137},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.38300000000000001,"15min":0.39000000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.49299999999999999,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48499999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.40200000000000002,"15min":0.40600000000000003},"min":{"1min":0.254,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.56299999999999994,"5min":1.111,"15min":1.111},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39500000000000002,"15min":0.39600000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.55700000000000005,"5min":1.016,"15min":1.016},"last":0.52800000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.53100000000000003,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.44700000000000001,"15min":0.45600000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.51700000000000002,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.51400000000000001}]},{"osd":4,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.46200000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.85399999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.627,"15min":0.627},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.86299999999999999,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.59699999999999998,"15min":0.59199999999999997},"min":{"1min":0.44900000000000001,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.878,"5min":1.228,"15min":1.228},"last":0.84299999999999997}]},{"osd":6,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.58999999999999997,"15min":0.58599999999999997},"min":{"1min":0.443,"5min":0.433,"15min":0.433},"max":{"1min":0.84399999999999997,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63100000000000001,"15min":0.629},"min":{"1min":0.48299999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.81200000000000006,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.73299999999999998}]},{"osd":7,"last update":"Wed Dec 13 23:57:05 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.64500000000000002,"15min":0.64300000000000002},"min":{"1min":0.46899999999999997,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.96699999999999997,"5min":1.454,"15min":1.454},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.60899999999999999,"15min":0.60499999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.87,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.877}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:57:56.980 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-13T23:55:01.135603+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=13, tm_hour=23, tm_min=55, tm_sec=1, tm_wday=2, tm_yday=347, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=13, tm_hour=23, tm_min=57, tm_sec=52, tm_wday=2, tm_yday=347, tm_isdst=0) 2023-12-13T23:57:56.981 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-13T23:58:12.408 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:12.411 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:16.982 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-13T23:58:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.093+0000 7f18233a9640 1 Processor -- start 2023-12-13T23:58:17.093 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.093+0000 7f18233a9640 1 -- start start 2023-12-13T23:58:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f18233a9640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7f181c150560 0x7f181c0b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f18233a9640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f181c150940 0x7f181c0ce0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f18233a9640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c0b49f0 con 0x7f181c0b3390 2023-12-13T23:58:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f18233a9640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0b3de0 con 0x7f181c150940 2023-12-13T23:58:17.096 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f18233a9640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0b40c0 con 0x7f181c150560 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f1821946640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f181c150940 0x7f181c0ce0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f1820944640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f181c0b3390 0x7f181c15d200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50212/0 (socket says 172.21.15.124:50212) 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.094+0000 7f1820944640 1 -- 172.21.15.124:0/614556136 learned_addr learned my addr 172.21.15.124:0/614556136 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821145640 1 --2- 172.21.15.124:0/614556136 >> v2:172.21.15.203:3300/0 conn(0x7f181c150560 0x7f181c0b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821946640 1 -- 172.21.15.124:0/614556136 >> v2:172.21.15.203:3300/0 conn(0x7f181c150560 msgr2=0x7f181c0b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821946640 1 --2- 172.21.15.124:0/614556136 >> v2:172.21.15.203:3300/0 conn(0x7f181c150560 0x7f181c0b2e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821946640 1 -- 172.21.15.124:0/614556136 >> v1:172.21.15.124:6789/0 conn(0x7f181c0b3390 legacy=0x7f181c15d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.097 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1813fff640 1 -- 172.21.15.124:0/614556136 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072218903 0 0) 0x7f181c0b49f0 con 0x7f181c0b3390 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821946640 1 -- 172.21.15.124:0/614556136 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0b43a0 con 0x7f181c150940 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.095+0000 7f1821946640 1 --2- 172.21.15.124:0/614556136 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150940 0x7f181c0ce0e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f181c072840 tx=0x7f181402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=540594f6a4d88f89 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f1813fff640 1 -- 172.21.15.124:0/614556136 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814031830 con 0x7f181c150940 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f1813fff640 1 -- 172.21.15.124:0/614556136 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814031990 con 0x7f181c150940 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f1813fff640 1 -- 172.21.15.124:0/614556136 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814031d20 con 0x7f181c150940 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 -- 172.21.15.124:0/614556136 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150940 msgr2=0x7f181c0ce0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 --2- 172.21.15.124:0/614556136 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150940 0x7f181c0ce0e0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f181c072840 tx=0x7f181402b8d0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.098 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 -- 172.21.15.124:0/614556136 shutdown_connections 2023-12-13T23:58:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 --2- 172.21.15.124:0/614556136 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150940 0x7f181c0ce0e0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 --2- 172.21.15.124:0/614556136 >> v2:172.21.15.203:3300/0 conn(0x7f181c150560 0x7f181c0b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 -- 172.21.15.124:0/614556136 >> 172.21.15.124:0/614556136 conn(0x7f181c06a4d0 msgr2=0x7f181c06a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 -- 172.21.15.124:0/614556136 shutdown_connections 2023-12-13T23:58:17.099 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.096+0000 7f18233a9640 1 -- 172.21.15.124:0/614556136 wait complete. 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.097+0000 7f18233a9640 1 Processor -- start 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.097+0000 7f18233a9640 1 -- start start 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f181c0b3390 0x7f181c1586d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f181c1551e0 0x7f181c146710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.102 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c152360 con 0x7f181c150560 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c15be30 con 0x7f181c0b3390 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f18233a9640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c15b870 con 0x7f181c1551e0 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f1820944640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f1820944640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:46424/0 (socket says 172.21.15.124:46424) 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f1820944640 1 -- 172.21.15.124:0/1370103189 learned_addr learned my addr 172.21.15.124:0/1370103189 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f1821145640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f181c0b3390 0x7f181c1586d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.098+0000 7f1821946640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f181c1551e0 0x7f181c146710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 -- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f181c1551e0 msgr2=0x7f181c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.103 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f181c1551e0 0x7f181c146710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 -- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f181c0b3390 msgr2=0x7f181c1586d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f181c0b3390 0x7f181c1586d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 -- 172.21.15.124:0/1370103189 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1814009400 con 0x7f181c150560 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1820944640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f180c00c4d0 tx=0x7f180c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c0074e0 con 0x7f181c150560 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c007660 con 0x7f181c150560 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c15bb50 con 0x7f181c150560 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180c004a20 con 0x7f181c150560 2023-12-13T23:58:17.104 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.099+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c147a50 con 0x7f181c150560 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.100+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c1524d0 con 0x7f181c150560 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.100+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f180c004b80 con 0x7f181c150560 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.101+0000 7f1811ffb640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 0x7f180006d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.101+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f180c088c30 con 0x7f181c150560 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.101+0000 7f1821145640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 0x7f180006d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.105 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.101+0000 7f1821145640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 0x7f180006d680 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f181800f510 tx=0x7f18180077b0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.106 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.105+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f180c053c80 con 0x7f181c150560 2023-12-13T23:58:17.270 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.270+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 --> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f181c073c80 con 0x7f180006b1c0 2023-12-13T23:58:17.272 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.273+0000 7f1811ffb640 1 -- 172.21.15.124:0/1370103189 <== mgr.4105 v2:172.21.15.203:6832/4285908483 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+51084 (secure 0 0 0) 0x7f181800d500 con 0x7f180006b1c0 2023-12-13T23:58:17.273 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:58:17.274 INFO:teuthology.orchestra.run.smithi124.stderr:dumped all 2023-12-13T23:58:17.276 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 msgr2=0x7f180006d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 0x7f180006d680 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f181800f510 tx=0x7f18180077b0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 msgr2=0x7f181c15adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f180c00c4d0 tx=0x7f180c00c990 comp rx=0 tx=0).stop 2023-12-13T23:58:17.277 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 shutdown_connections 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f180006b1c0 0x7f180006d680 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f181c1551e0 0x7f181c146710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f181c150560 0x7f181c15adb0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 --2- 172.21.15.124:0/1370103189 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f181c0b3390 0x7f181c1586d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.274+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 >> 172.21.15.124:0/1370103189 conn(0x7f181c06a4d0 msgr2=0x7f181c0cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.275+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 shutdown_connections 2023-12-13T23:58:17.278 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.275+0000 7f18233a9640 1 -- 172.21.15.124:0/1370103189 wait complete. 2023-12-13T23:58:17.285 INFO:teuthology.orchestra.run.smithi124.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2023-12-13T23:58:16.153566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":226464,"kb_used_data":6152,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754748256,"statfs":{"total":773094113280,"available":772862214144,"internally_reserved":0,"allocated":6299648,"data_stored":3294552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12756,"internal_metadata":225496620},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002294"},"pg_stats":[{"pgid":"1.0","version":"53'119","reported_seq":209,"reported_epoch":53,"state":"active+clean","last_fresh":"2023-12-13T23:57:58.382848+0000","last_change":"2023-12-13T23:57:58.382878+0000","last_active":"2023-12-13T23:57:58.382848+0000","last_peered":"2023-12-13T23:57:58.382848+0000","last_clean":"2023-12-13T23:57:58.382848+0000","last_became_active":"2023-12-13T23:55:02.357062+0000","last_became_peered":"2023-12-13T23:55:02.357062+0000","last_unstale":"2023-12-13T23:57:58.382848+0000","last_undegraded":"2023-12-13T23:57:58.382848+0000","last_fullsized":"2023-12-13T23:57:58.382848+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"53'119","last_scrub_stamp":"2023-12-13T23:57:58.382763+0000","last_deep_scrub":"53'119","last_deep_scrub_stamp":"2023-12-13T23:57:58.382763+0000","last_clean_scrub_stamp":"2023-12-13T23:57:58.382763+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-15T01:21:06.508515+0000","scrub_duration":0.004793907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32380,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339460,"statfs":{"total":96636764160,"available":96603607040,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52700000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.68100000000000005,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77500000000000002,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63400000000000001,"5min":0.754,"15min":0.754},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55400000000000005,"15min":0.55600000000000005},"min":{"1min":0.42899999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.70199999999999996,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.57799999999999996}]},{"osd":2,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54500000000000004,"15min":0.54100000000000004},"min":{"1min":0.437,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.64400000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.55200000000000005,"15min":0.55700000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71999999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.52800000000000002,"15min":0.53000000000000003},"min":{"1min":0.41999999999999998,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70599999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.59599999999999997,"15min":0.59999999999999998},"min":{"1min":0.42999999999999999,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.85099999999999998,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.66800000000000004}]},{"osd":4,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46100000000000002,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.52200000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68400000000000005,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.442}]},{"osd":5,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55400000000000005,"15min":0.57199999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.752,"5min":1.194,"15min":1.194},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.56599999999999995,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81399999999999995,"5min":1.002,"15min":1.002},"last":0.45500000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:58:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.40699999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84999999999999998,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.621},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.60099999999999998,"15min":0.61599999999999999},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76600000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.47499999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55800000000000005,"15min":0.55600000000000005},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.76500000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.56899999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68100000000000005,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.58499999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.63500000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.748,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.55600000000000005}]},{"osd":2,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.69599999999999995,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59599999999999997,"15min":0.60199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68899999999999995,"5min":1.006,"15min":1.006},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70899999999999996,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.65700000000000003,"15min":0.69899999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74399999999999999,"5min":3.5190000000000001,"15min":3.5190000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.34300000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71399999999999997,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.53400000000000003,"15min":0.54500000000000004},"min":{"1min":0.32100000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72799999999999998,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47699999999999998}]},{"osd":5,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.376,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.68500000000000005,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.58199999999999996,"15min":0.59699999999999998},"min":{"1min":0.36799999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.434}]},{"osd":7,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.59299999999999997,"15min":0.61199999999999999},"min":{"1min":0.35699999999999998,"5min":0.307,"15min":0.307},"max":{"1min":0.78000000000000003,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.60099999999999998,"15min":0.62},"min":{"1min":0.42599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.72999999999999998,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.502}]}]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.52600000000000002,"15min":0.50800000000000001},"min":{"1min":0.42799999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.41899999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.90200000000000002,"5min":1.03,"15min":1.03},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.53400000000000003,"15min":0.51600000000000001},"min":{"1min":0.39300000000000002,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.57599999999999996,"15min":0.56599999999999995},"min":{"1min":0.41799999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81599999999999995,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.58299999999999996,"15min":0.56000000000000005},"min":{"1min":0.55300000000000005,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.56299999999999994,"15min":0.55500000000000005},"min":{"1min":0.44600000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58999999999999997,"15min":0.57899999999999996},"min":{"1min":0.40300000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85899999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.56200000000000006,"15min":0.53900000000000003},"min":{"1min":0.441,"5min":0.376,"15min":0.376},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.66000000000000003}]},{"osd":4,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.33700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.95699999999999996,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.73499999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52300000000000002,"15min":0.51100000000000001},"min":{"1min":0.36399999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.43099999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.625}]},{"osd":7,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.59299999999999997,"15min":0.58899999999999997},"min":{"1min":0.41299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.60199999999999998,"15min":0.58699999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.755}]}]},{"osd":4,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.51900000000000002,"15min":0.51100000000000001},"min":{"1min":0.34200000000000003,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.65100000000000002,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56599999999999995}]},{"osd":1,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.54300000000000004,"15min":0.54100000000000004},"min":{"1min":0.42299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.54700000000000004,"15min":0.53800000000000003},"min":{"1min":0.38500000000000001,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78600000000000003,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.47699999999999998}]},{"osd":2,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54600000000000004,"15min":0.54500000000000004},"min":{"1min":0.41799999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.60799999999999998}]},{"osd":3,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72699999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56599999999999995},"min":{"1min":0.42399999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.76900000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.59199999999999997}]},{"osd":5,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.52500000000000002,"15min":0.52200000000000002},"min":{"1min":0.33300000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73199999999999998,"5min":0.748,"15min":0.748},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.36799999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.627,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.42399999999999999}]},{"osd":6,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.58799999999999997,"15min":0.59599999999999997},"min":{"1min":0.39900000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.70899999999999996,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.54700000000000004}]},{"osd":7,"last update":"Wed Dec 13 23:58:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.67900000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71699999999999997,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.67400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574891,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.32000000000000001,"15min":0.32400000000000001},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.40500000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.32100000000000001},"min":{"1min":0.254,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.41899999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"last":0.29499999999999998}]},{"osd":1,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.32600000000000001,"5min":0.33200000000000002,"15min":0.32800000000000001},"min":{"1min":0.25,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52900000000000003,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.34699999999999998,"15min":0.34100000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.42399999999999999,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.32300000000000001}]},{"osd":2,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.37,"15min":0.36699999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.46100000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.376},{"interface":"front","average":{"1min":0.377,"5min":0.39400000000000002,"15min":0.39200000000000002},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.46700000000000003,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.70799999999999996,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55800000000000005,"15min":0.55200000000000005},"min":{"1min":0.39000000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.78100000000000003,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.42299999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59499999999999997,"15min":0.59299999999999997},"min":{"1min":0.42899999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61899999999999999,"15min":0.625},"min":{"1min":0.47699999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.70499999999999996,"5min":1.038,"15min":1.038},"last":0.52000000000000002}]},{"osd":6,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.60499999999999998,"15min":0.61299999999999999},"min":{"1min":0.443,"5min":0.434,"15min":0.434},"max":{"1min":0.74199999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.45300000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73799999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.46200000000000002}]},{"osd":7,"last update":"Wed Dec 13 23:58:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.58099999999999996,"15min":0.57599999999999996},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67900000000000005,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.433},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.441,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.76200000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.57099999999999995}]}]},{"osd":2,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.29199999999999998,"5min":0.315,"15min":0.317},"min":{"1min":0.24299999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.375,"5min":0.55700000000000005,"15min":0.55700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.32300000000000001,"15min":0.32000000000000001},"min":{"1min":0.215,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.41199999999999998,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.48699999999999999}]},{"osd":1,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.33800000000000002,"15min":0.33700000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.39300000000000002,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36499999999999999,"15min":0.35899999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.51000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33200000000000002,"15min":0.32000000000000001},"min":{"1min":0.26800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.45400000000000001,"5min":0.63,"15min":0.63},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.375,"15min":0.372},"min":{"1min":0.316,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.438,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58299999999999996}]},{"osd":4,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.57999999999999996,"15min":0.56799999999999995},"min":{"1min":0.41799999999999998,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77600000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.57899999999999996},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.88600000000000001}]},{"osd":5,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.57599999999999996,"15min":0.57799999999999996},"min":{"1min":0.40999999999999998,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76200000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60299999999999998,"15min":0.59999999999999998},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.87,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.90800000000000003}]},{"osd":6,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56299999999999994,"15min":0.56000000000000005},"min":{"1min":0.40000000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.79700000000000004,"5min":0.871,"15min":0.871},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.83999999999999997,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76200000000000001}]},{"osd":7,"last update":"Wed Dec 13 23:58:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.58199999999999996,"15min":0.58599999999999997},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.70699999999999996,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.79000000000000004,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.83699999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27644,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":638976,"data_stored":264227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.38600000000000001,"15min":0.40500000000000003},"min":{"1min":0.245,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47599999999999998,"5min":1.488,"15min":1.488},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.32300000000000001,"5min":0.36499999999999999,"15min":0.373},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.53200000000000003,"5min":1.3520000000000001,"15min":1.3520000000000001},"last":0.503}]},{"osd":2,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.374,"15min":0.39500000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.42599999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.39000000000000001,"15min":0.39400000000000002},"min":{"1min":0.26800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57599999999999996,"5min":1.536,"15min":1.536},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.42499999999999999,"15min":0.441},"min":{"1min":0.29199999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54500000000000004,"5min":1.512,"15min":1.512},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.505,"5min":1.401,"15min":1.401},"last":0.40300000000000002}]},{"osd":4,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59799999999999998,"15min":0.60099999999999998},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.79000000000000004,"5min":1.6379999999999999,"15min":1.6379999999999999},"last":0.623},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.63500000000000001},"min":{"1min":0.44400000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80900000000000005,"5min":1.4339999999999999,"15min":1.4339999999999999},"last":0.66900000000000004}]},{"osd":5,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56399999999999995,"15min":0.56799999999999995},"min":{"1min":0.48299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.749,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.755},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58899999999999997,"15min":0.60499999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68000000000000005,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.68200000000000005}]},{"osd":6,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.59599999999999997},"min":{"1min":0.44800000000000001,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77300000000000002,"5min":1.292,"15min":1.292},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.42199999999999999,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.78800000000000003,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.59299999999999997}]},{"osd":7,"last update":"Wed Dec 13 23:58:03 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60999999999999999,"15min":0.61599999999999999},"min":{"1min":0.45600000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.73599999999999999,"5min":1.46,"15min":1.46},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.624,"15min":0.627},"min":{"1min":0.48499999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76900000000000002,"5min":1.377,"15min":1.377},"last":0.73399999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28220,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343620,"statfs":{"total":96636764160,"available":96607866880,"internally_reserved":0,"allocated":1232896,"data_stored":854595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.39200000000000002,"15min":0.41099999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.73199999999999998,"5min":1.137,"15min":1.137},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.377,"15min":0.38800000000000001},"min":{"1min":0.223,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.504,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.30499999999999999}]},{"osd":2,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.39600000000000002,"15min":0.40400000000000003},"min":{"1min":0.26200000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.63300000000000001,"5min":1.111,"15min":1.111},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.39100000000000001,"15min":0.39500000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.61699999999999999,"5min":1.016,"15min":1.016},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.59699999999999998,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.372},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.432,"15min":0.45100000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.58799999999999997,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.40699999999999997}]},{"osd":4,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.40699999999999997,"5min":0.376,"15min":0.376},"max":{"1min":0.81200000000000006,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57399999999999995,"15min":0.56399999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.86899999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61699999999999999,"15min":0.623},"min":{"1min":0.40799999999999997,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.79300000000000004,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59899999999999998,"15min":0.59299999999999997},"min":{"1min":0.47899999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.84299999999999997,"5min":1.228,"15min":1.228},"last":0.56899999999999995}]},{"osd":6,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58899999999999997,"15min":0.58599999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80400000000000005,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.628,"15min":0.628},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.79400000000000004,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.68000000000000005}]},{"osd":7,"last update":"Wed Dec 13 23:58:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.63900000000000001,"15min":0.64100000000000001},"min":{"1min":0.441,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76200000000000001,"5min":1.454,"15min":1.454},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.60599999999999998,"15min":0.60399999999999998},"min":{"1min":0.442,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.877,"5min":1.4139999999999999,"15min":1.4139999999999999},"last":0.58399999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-13T23:58:17.289 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-13T23:58:17.389 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.390+0000 7fba7a25c640 1 Processor -- start 2023-12-13T23:58:17.390 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.390+0000 7fba7a25c640 1 -- start start 2023-12-13T23:58:17.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba7a25c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba740b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba7a25c640 1 --2- >> v2:172.21.15.203:3300/0 conn(0x7fba7415e730 0x7fba740b1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba7a25c640 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba7415e2a0 con 0x7fba7415a4e0 2023-12-13T23:58:17.393 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba7a25c640 1 -- --> v2:172.21.15.203:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba7415b520 con 0x7fba7415e730 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba7a25c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba74143870 con 0x7fba74143490 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba73fff640 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fba7415a4e0 0x7fba7415a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:50228/0 (socket says 172.21.15.124:50228) 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba73fff640 1 -- 172.21.15.124:0/2382652670 learned_addr learned my addr 172.21.15.124:0/2382652670 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569765149 0 0) 0x7fba7415e2a0 con 0x7fba7415a4e0 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba5c003520 con 0x7fba7415a4e0 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba72ffd640 1 --2- 172.21.15.124:0/2382652670 >> v2:172.21.15.203:3300/0 conn(0x7fba7415e730 0x7fba740b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.397 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba737fe640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba740b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 137212441 0 0) 0x7fba5c003520 con 0x7fba7415a4e0 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 msgr2=0x7fba740b0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba740b0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 >> v2:172.21.15.203:3300/0 conn(0x7fba7415e730 msgr2=0x7fba740b1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 --2- 172.21.15.124:0/2382652670 >> v2:172.21.15.203:3300/0 conn(0x7fba7415e730 0x7fba740b1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba7415b240 con 0x7fba7415a4e0 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3984989457 0 0) 0x7fba6c0019f0 con 0x7fba7415a4e0 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 >> v1:172.21.15.124:6789/0 conn(0x7fba7415a4e0 legacy=0x7fba7415a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.398 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba5c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba5c003fa0 0x7fba5c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c004d00 0x7fba5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7415b520 con 0x7fba5c003fa0 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74143870 con 0x7fba5c004d00 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c00ba80 con 0x7fba74143490 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.391+0000 7fba72ffd640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba5c003fa0 0x7fba5c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba737fe640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba5c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.399 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba73fff640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c004d00 0x7fba5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 -- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 msgr2=0x7fba5c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba74143490 0x7fba5c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 -- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c004d00 msgr2=0x7fba5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba5c004d00 0x7fba5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 -- 172.21.15.124:0/2382652670 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c003520 con 0x7fba5c003fa0 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.392+0000 7fba72ffd640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba5c003fa0 0x7fba5c0047c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba6400ba30 tx=0x7fba5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ece0dfa18f0eba server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6402e970 con 0x7fba5c003fa0 2023-12-13T23:58:17.400 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba64032070 con 0x7fba5c003fa0 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba727fc640 1 -- 172.21.15.124:0/2382652670 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6402ecf0 con 0x7fba5c003fa0 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 -- 172.21.15.124:0/2382652670 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba5c003fa0 msgr2=0x7fba5c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 --2- 172.21.15.124:0/2382652670 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba5c003fa0 0x7fba5c0047c0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fba6400ba30 tx=0x7fba5c0057f0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba73fff640 1 -- 172.21.15.124:0/2382652670 reap_dead start 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 -- 172.21.15.124:0/2382652670 shutdown_connections 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 -- 172.21.15.124:0/2382652670 >> 172.21.15.124:0/2382652670 conn(0x7fba7406a4d0 msgr2=0x7fba7406a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 -- 172.21.15.124:0/2382652670 shutdown_connections 2023-12-13T23:58:17.401 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.393+0000 7fba7a25c640 1 -- 172.21.15.124:0/2382652670 wait complete. 2023-12-13T23:58:17.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.394+0000 7fba7a25c640 1 Processor -- start 2023-12-13T23:58:17.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 -- start start 2023-12-13T23:58:17.404 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba5c003fa0 0x7fba740ab590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba7412ff90 0x7fba74141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba741304f0 con 0x7fba740abad0 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba741307d0 con 0x7fba7412ff90 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba7a25c640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba740ad480 con 0x7fba5c003fa0 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba72ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba72ffd640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:46446/0 (socket says 172.21.15.124:46446) 2023-12-13T23:58:17.405 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba72ffd640 1 -- 172.21.15.124:0/225419050 learned_addr learned my addr 172.21.15.124:0/225419050 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.395+0000 7fba73fff640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba7412ff90 0x7fba74141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba737fe640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba5c003fa0 0x7fba740ab590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 -- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba5c003fa0 msgr2=0x7fba740ab590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba5c003fa0 0x7fba740ab590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 -- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba7412ff90 msgr2=0x7fba74141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba7412ff90 0x7fba74141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 -- 172.21.15.124:0/225419050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba740ad760 con 0x7fba740abad0 2023-12-13T23:58:17.406 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba72ffd640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fba6400bb50 tx=0x7fba64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba64039c50 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba64039dd0 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7413ed40 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba640436b0 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.396+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7412d9c0 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.397+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba740ad8f0 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.397+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7fba64043810 con 0x7fba740abad0 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.398+0000 7fba5bfff640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 0x7fba6006d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.407 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.398+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fba640ae0d0 con 0x7fba740abad0 2023-12-13T23:58:17.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.398+0000 7fba737fe640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 0x7fba6006d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.398+0000 7fba737fe640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 0x7fba6006d6b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fba68006fd0 tx=0x7fba68008040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.408 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.402+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7fba64079070 con 0x7fba740abad0 2023-12-13T23:58:17.570 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.571+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fba7406ade0 con 0x7fba740abad0 2023-12-13T23:58:17.573 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.574+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) v1 ==== 123+0+0 (secure 0 0 0) 0x7fba6407c690 con 0x7fba740abad0 2023-12-13T23:58:17.575 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.575+0000 7fba5bfff640 1 -- 172.21.15.124:0/225419050 <== mon.0 v2:172.21.15.124:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fba64079070 con 0x7fba740abad0 2023-12-13T23:58:17.576 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 msgr2=0x7fba6006d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 0x7fba6006d6b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fba68006fd0 tx=0x7fba68008040 comp rx=0 tx=0).stop 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 msgr2=0x7fba7412fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fba6400bb50 tx=0x7fba64004730 comp rx=0 tx=0).stop 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 shutdown_connections 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7fba6006b1f0 0x7fba6006d6b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7fba7412ff90 0x7fba74141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.577 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba740abad0 0x7fba7412fa50 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 --2- 172.21.15.124:0/225419050 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7fba5c003fa0 0x7fba740ab590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 >> 172.21.15.124:0/225419050 conn(0x7fba7406a4d0 msgr2=0x7fba740afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 shutdown_connections 2023-12-13T23:58:17.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.577+0000 7fba7a25c640 1 -- 172.21.15.124:0/225419050 wait complete. 2023-12-13T23:58:17.587 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-12-13T23:58:17.587 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-12-13T23:58:17.687 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.687+0000 7f6abf525640 1 Processor -- start 2023-12-13T23:58:17.687 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 -- start start 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab815d330 0x7f6ab80b8b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6ab80d2910 0x7f6ab80d31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab80b2b10 con 0x7f6ab80d2d90 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab80d1590 con 0x7f6ab80d2910 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.688+0000 7f6abf525640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80d1870 con 0x7f6ab815d330 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6ab80d2910 0x7f6ab80d31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.689 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6ab80d2910 0x7f6ab80d31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:46462/0 (socket says 172.21.15.124:46462) 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 -- 172.21.15.124:0/879032678 learned_addr learned my addr 172.21.15.124:0/879032678 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abd2c1640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab815d330 0x7f6ab80b8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 -- 172.21.15.124:0/879032678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab815d330 msgr2=0x7f6ab80b8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab815d330 0x7f6ab80b8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 -- 172.21.15.124:0/879032678 >> v1:172.21.15.203:6789/0 conn(0x7f6ab80d2d90 legacy=0x7f6ab80b6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.690 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.689+0000 7f6abcac0640 1 -- 172.21.15.124:0/879032678 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab80d1b50 con 0x7f6ab80d2910 2023-12-13T23:58:17.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abcac0640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab80d2910 0x7f6ab80d31f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009870 tx=0x7f6aa802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e19c2a102201db server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6aaffff640 1 -- 172.21.15.124:0/879032678 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8031640 con 0x7f6ab80d2910 2023-12-13T23:58:17.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6aaffff640 1 -- 172.21.15.124:0/879032678 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6aa80317a0 con 0x7f6ab80d2910 2023-12-13T23:58:17.693 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6aaffff640 1 -- 172.21.15.124:0/879032678 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa8031b50 con 0x7f6ab80d2910 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 -- 172.21.15.124:0/879032678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab80d2910 msgr2=0x7f6ab80d31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab80d2910 0x7f6ab80d31f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f6aa8009870 tx=0x7f6aa802b8f0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 -- 172.21.15.124:0/879032678 shutdown_connections 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab80d2910 0x7f6ab80d31f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 --2- 172.21.15.124:0/879032678 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab815d330 0x7f6ab80b8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.690+0000 7f6abf525640 1 -- 172.21.15.124:0/879032678 >> 172.21.15.124:0/879032678 conn(0x7f6ab8072020 msgr2=0x7f6ab8072430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.691+0000 7f6abf525640 1 -- 172.21.15.124:0/879032678 shutdown_connections 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.691+0000 7f6abf525640 1 -- 172.21.15.124:0/879032678 wait complete. 2023-12-13T23:58:17.694 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 Processor -- start 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 -- start start 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 --2- >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab80d2910 0x7f6ab80c9780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 --2- >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 0x7f6ab8152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81f3410 con 0x7f6ab815d330 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 -- --> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab80cb090 con 0x7f6ab80d2d90 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abf525640 1 -- --> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab81f36d0 con 0x7f6ab80d2910 2023-12-13T23:58:17.695 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.692+0000 7f6abdac2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:46464/0 (socket says 172.21.15.124:46464) 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 -- 172.21.15.124:0/3560905341 learned_addr learned my addr 172.21.15.124:0/3560905341 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abcac0640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 0x7f6ab8152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abd2c1640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab80d2910 0x7f6ab80c9780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 -- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab80d2910 msgr2=0x7f6ab80c9780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab80d2910 0x7f6ab80c9780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 -- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 msgr2=0x7f6ab8152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 0x7f6ab8152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.696 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 -- 172.21.15.124:0/3560905341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa80094d0 con 0x7f6ab815d330 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abcac0640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 0x7f6ab8152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abdac2640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000c3a0 tx=0x7f6ab000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab00052d0 con 0x7f6ab815d330 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f6ab0005430 con 0x7f6ab815d330 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab81f39b0 con 0x7f6ab815d330 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0003850 con 0x7f6ab815d330 2023-12-13T23:58:17.697 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.693+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab80cacd0 con 0x7f6ab815d330 2023-12-13T23:58:17.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.694+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab80cae60 con 0x7f6ab815d330 2023-12-13T23:58:17.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.695+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 89370+0+0 (secure 0 0 0) 0x7f6ab00039b0 con 0x7f6ab815d330 2023-12-13T23:58:17.698 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.695+0000 7f6aadffb640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 0x7f6a9c06d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-13T23:58:17.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.695+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f6ab0088ca0 con 0x7f6ab815d330 2023-12-13T23:58:17.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.695+0000 7f6abd2c1640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 0x7f6a9c06d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-13T23:58:17.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.696+0000 7f6abd2c1640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 0x7f6a9c06d680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f60 tx=0x7f6ab400b040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-13T23:58:17.699 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.699+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179831 (secure 0 0 0) 0x7f6ab0053d90 con 0x7f6ab815d330 2023-12-13T23:58:17.901 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.902+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab81f3ed0 con 0x7f6ab815d330 2023-12-13T23:58:17.902 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.902+0000 7f6aadffb640 1 -- 172.21.15.124:0/3560905341 <== mon.0 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+3708 (secure 0 0 0) 0x7f6ab81f3ed0 con 0x7f6ab815d330 2023-12-13T23:58:17.902 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-12-13T23:58:17.902 INFO:teuthology.orchestra.run.smithi124.stdout:{"epoch":24,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5488,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6836/2935465896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6834","nonce":2935465896},{"type":"v1","addr":"172.21.15.203:6836","nonce":2935465896}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5782,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.203:6837/943139476","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.203:6835","nonce":943139476},{"type":"v1","addr":"172.21.15.203:6837","nonce":943139476}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":24},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6833/2159743524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":2159743524},{"type":"v1","addr":"172.21.15.124:6833","nonce":2159743524}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5901,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/2003165619","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":2003165619},{"type":"v1","addr":"172.21.15.124:6835","nonce":2003165619}]},"join_fscid":-1,"export_targets":[],"features":4540701272860131327,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-12-13T23:58:17.903 INFO:teuthology.orchestra.run.smithi124.stderr:dumped fsmap epoch 24 2023-12-13T23:58:17.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 msgr2=0x7f6a9c06d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 0x7f6a9c06d680 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f60 tx=0x7f6ab400b040 comp rx=0 tx=0).stop 2023-12-13T23:58:17.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 msgr2=0x7f6ab81560f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-13T23:58:17.904 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000c3a0 tx=0x7f6ab000c860 comp rx=0 tx=0).stop 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 shutdown_connections 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:6832/4285908483,v1:172.21.15.203:6833/4285908483] conn(0x7f6a9c06b1c0 0x7f6a9c06d680 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3300/0,v1:172.21.15.203:6789/0] conn(0x7f6ab80d2d90 0x7f6ab8152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6ab815d330 0x7f6ab81560f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 --2- 172.21.15.124:0/3560905341 >> [v2:172.21.15.203:3301/0,v1:172.21.15.203:6790/0] conn(0x7f6ab80d2910 0x7f6ab80c9780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 >> 172.21.15.124:0/3560905341 conn(0x7f6ab8072020 msgr2=0x7f6ab80d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 shutdown_connections 2023-12-13T23:58:17.905 INFO:teuthology.orchestra.run.smithi124.stderr:2023-12-13T23:58:17.904+0000 7f6abf525640 1 -- 172.21.15.124:0/3560905341 wait complete. 2023-12-13T23:58:17.915 DEBUG:tasks.cephfs.filesystem:already dead... 2023-12-13T23:58:17.915 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-13T23:58:17.915 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-12-13T23:58:17.915 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:17.924 INFO:tasks.ceph.mds.a:Stopped 2023-12-13T23:58:17.925 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-12-13T23:58:17.925 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:17.977 INFO:tasks.ceph.mds.c:Stopped 2023-12-13T23:58:17.978 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-12-13T23:58:17.978 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:17.990 INFO:tasks.ceph.mds.b:Stopped 2023-12-13T23:58:17.990 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-12-13T23:58:17.991 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.042 INFO:tasks.ceph.mds.d:Stopped 2023-12-13T23:58:18.043 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-13T23:58:18.043 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-13T23:58:18.043 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.076 INFO:tasks.ceph.osd.0:Stopped 2023-12-13T23:58:18.076 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-13T23:58:18.076 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.148 INFO:tasks.ceph.osd.1:Stopped 2023-12-13T23:58:18.149 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-13T23:58:18.149 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.217 INFO:tasks.ceph.osd.2:Stopped 2023-12-13T23:58:18.218 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-13T23:58:18.218 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.281 INFO:tasks.ceph.osd.3:Stopped 2023-12-13T23:58:18.281 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-13T23:58:18.281 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.310 INFO:tasks.ceph.osd.4:Stopped 2023-12-13T23:58:18.311 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-13T23:58:18.311 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.385 INFO:tasks.ceph.osd.5:Stopped 2023-12-13T23:58:18.386 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-13T23:58:18.386 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.451 INFO:tasks.ceph.osd.6:Stopped 2023-12-13T23:58:18.451 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-13T23:58:18.451 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.518 INFO:tasks.ceph.osd.7:Stopped 2023-12-13T23:58:18.519 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-13T23:58:18.519 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-13T23:58:18.519 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.544 INFO:tasks.ceph.mgr.y:Stopped 2023-12-13T23:58:18.544 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-13T23:58:18.544 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.586 INFO:tasks.ceph.mgr.x:Stopped 2023-12-13T23:58:18.586 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-13T23:58:18.586 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-13T23:58:18.586 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.597 INFO:tasks.ceph.mon.a:Stopped 2023-12-13T23:58:18.598 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-13T23:58:18.598 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.641 INFO:tasks.ceph.mon.b:Stopped 2023-12-13T23:58:18.641 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-13T23:58:18.641 INFO:teuthology.orchestra.run:waiting for 300 2023-12-13T23:58:18.693 INFO:tasks.ceph.mon.c:Stopped 2023-12-13T23:58:18.693 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-13T23:58:18.694 DEBUG:teuthology.orchestra.run.smithi124:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-13T23:58:18.725 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:58:18.725 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-13T23:58:18.945 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:58:18.945 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-13T23:58:19.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:58:19.057 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-13T23:58:19.106 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:58:19.106 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-13T23:58:19.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:58:19.157 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-13T23:58:19.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:58:19.304 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-13T23:58:19.360 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:58:19.360 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-13T23:58:19.416 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:58:19.417 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-13T23:58:19.476 INFO:tasks.ceph:Archiving mon data... 2023-12-13T23:58:19.477 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/data/mon.a.tgz 2023-12-13T23:58:19.478 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-12-13T23:58:19.494 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.jLzQMwu97J 2023-12-13T23:58:19.494 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.jLzQMwu97J 2023-12-13T23:58:19.635 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.jLzQMwu97J 2023-12-13T23:58:19.719 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.jLzQMwu97J is 341KB 2023-12-13T23:58:19.816 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.jLzQMwu97J 2023-12-13T23:58:19.831 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/data/mon.b.tgz 2023-12-13T23:58:19.831 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2023-12-13T23:58:19.848 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.BZ1YX19jbs 2023-12-13T23:58:19.849 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.BZ1YX19jbs 2023-12-13T23:58:20.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.BZ1YX19jbs 2023-12-13T23:58:20.102 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.BZ1YX19jbs is 445KB 2023-12-13T23:58:20.199 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.BZ1YX19jbs 2023-12-13T23:58:20.214 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/data/mon.c.tgz 2023-12-13T23:58:20.214 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2023-12-13T23:58:20.268 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.uPP0s2qNfB 2023-12-13T23:58:20.269 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.uPP0s2qNfB 2023-12-13T23:58:20.419 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.uPP0s2qNfB 2023-12-13T23:58:20.503 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.uPP0s2qNfB is 445KB 2023-12-13T23:58:20.603 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.uPP0s2qNfB 2023-12-13T23:58:20.617 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-13T23:58:20.617 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-13T23:58:20.620 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-13T23:58:20.702 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-13T23:58:20.739 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-13T23:58:20.739 INFO:tasks.ceph:Archiving crash dumps... 2023-12-13T23:58:20.741 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi124/crash 2023-12-13T23:58:20.742 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-13T23:58:20.770 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi203/crash 2023-12-13T23:58:20.771 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-13T23:58:20.805 INFO:tasks.ceph:Shutting down logrotate 2023-12-13T23:58:20.806 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:20.816 DEBUG:teuthology.orchestra.run.smithi203:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:20.928 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:20.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-12-13T23:58:20.983 INFO:tasks.ceph:Compressing logs... 2023-12-13T23:58:20.983 DEBUG:teuthology.orchestra.run.smithi124:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-13T23:58:21.000 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-13T23:58:21.086 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-13T23:58:21.086 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89023.log 2023-12-13T23:58:21.086 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88863.log 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.89023.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88863.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.89023.log.gz 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.88863.log.gz 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89063.log 2023-12-13T23:58:21.087 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.89063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.95762.log 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.89063.log.gz 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88783.log 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.3.95762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88943.log 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88783.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.88783.log.gz 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88943.log: 90.3% -- replaced with /var/log/ceph/ceph-client.3.95762.log.gz 2023-12-13T23:58:21.088 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.7.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.88943.log.gz 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88903.log 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88903.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88903.log.gz/var/log/ceph/ceph-mgr.x.log: 2023-12-13T23:58:21.089 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88983.log 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: 91.9% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88823.log 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88983.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.88983.log.gz 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.95316.log 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.88823.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.88823.log.gz 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.1.95316.log: 89.5% -- replaced with /var/log/ceph/ceph-client.1.95316.log.gz 2023-12-13T23:58:21.090 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-13T23:58:21.091 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 90.7% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-12-13T23:58:21.092 INFO:teuthology.orchestra.run.smithi203.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-12-13T23:58:21.128 INFO:teuthology.orchestra.run.smithi203.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-13T23:58:21.186 INFO:teuthology.orchestra.run.smithi203.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-13T23:58:21.191 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102414.log 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105279.log 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96948.log 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96160.log 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102414.log.gz 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96948.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105279.log.gz 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105363.log 2023-12-13T23:58:21.192 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96160.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96948.log.gz 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97158.log 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105363.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.96160.log.gz 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101130.log 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97158.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105363.log.gz 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95436.log 2023-12-13T23:58:21.193 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101130.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-13T23:58:21.194 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97158.log.gz 2023-12-13T23:58:21.194 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101130.log.gz 2023-12-13T23:58:21.194 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95825.log 2023-12-13T23:58:21.194 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100206.log 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95436.log.gz 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100458.log 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95825.log.gz 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99429.log 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100458.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100206.log.gz 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99075.log 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99429.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100458.log.gz 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109128.log 2023-12-13T23:58:21.195 INFO:teuthology.orchestra.run.smithi124.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.99429.log.gz 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101046.log 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109128.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99075.log.gz 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101592.log 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101046.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109128.log.gz 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95478.log 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.101046.log.gz 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95520.log 2023-12-13T23:58:21.196 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95100.log 2023-12-13T23:58:21.197 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.101592.log.gz 2023-12-13T23:58:21.197 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95520.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.95478.log.gz 2023-12-13T23:58:21.197 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111158.log 2023-12-13T23:58:21.197 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95520.log.gz 2023-12-13T23:58:21.197 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98739.log 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102762.log 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.95100.log.gz 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98739.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111158.log.gz 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100884.log 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98739.log.gz 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111664.log 2023-12-13T23:58:21.198 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100884.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102762.log.gz 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96738.log 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111664.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.100884.log.gz -5 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.109070.log 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111664.log.gz 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94576.log 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100374.log 2023-12-13T23:58:21.199 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109070.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96738.log.gz 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.94576.log: -- /var/log/ceph/ceph-client.admin.102162.log 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109070.log.gz 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100374.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94576.log.gz 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98461.log 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102162.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100374.log.gz 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109438.log 2023-12-13T23:58:21.200 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98461.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102162.log.gz 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99345.log 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98461.log.gz 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99834.log 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99345.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109438.log.gz 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96780.log 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr: 89.4%/var/log/ceph/ceph-client.admin.99834.log: -- replaced with /var/log/ceph/ceph-client.admin.99345.log.gz 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96822.log 2023-12-13T23:58:21.201 INFO:teuthology.orchestra.run.smithi124.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.99834.log.gz 2023-12-13T23:58:21.202 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111874.log 2023-12-13T23:58:21.202 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96822.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96780.log.gz 2023-12-13T23:58:21.202 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112910.log 2023-12-13T23:58:21.202 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111874.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96822.log.gz 2023-12-13T23:58:21.203 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111370.log 2023-12-13T23:58:21.203 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.111874.log.gz 2023-12-13T23:58:21.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109600.log 2023-12-13T23:58:21.203 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111370.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.112910.log.gz -5 2023-12-13T23:58:21.204 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.102498.log 2023-12-13T23:58:21.204 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109600.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111370.log.gz 2023-12-13T23:58:21.204 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99159.log 2023-12-13T23:58:21.204 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109600.log.gz 2023-12-13T23:58:21.205 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98865.log 2023-12-13T23:58:21.205 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102498.log: /var/log/ceph/ceph-client.admin.99159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112448.log 2023-12-13T23:58:21.205 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102498.log.gz 2023-12-13T23:58:21.205 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98865.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.99159.log.gz 2023-12-13T23:58:21.206 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95604.log 2023-12-13T23:58:21.206 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.98865.log.gz 2023-12-13T23:58:21.206 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112826.log 2023-12-13T23:58:21.206 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95604.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112448.log.gz 2023-12-13T23:58:21.207 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.109859.log 2023-12-13T23:58:21.207 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112826.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95604.log.gz 2023-12-13T23:58:21.207 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95184.log 2023-12-13T23:58:21.207 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.109859.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.112826.log.gz 2023-12-13T23:58:21.208 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100122.log 2023-12-13T23:58:21.208 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112406.log 2023-12-13T23:58:21.208 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100122.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95184.log.gz 2023-12-13T23:58:21.209 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112742.log 2023-12-13T23:58:21.209 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100122.log.gz 2023-12-13T23:58:21.209 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97326.log 2023-12-13T23:58:21.209 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.112406.log.gz 2023-12-13T23:58:21.210 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104054.log 2023-12-13T23:58:21.210 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112742.log: 89.9% -- replaced with /var/log/ceph/ceph-client.0.109859.log.gz 2023-12-13T23:58:21.210 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106075.log 2023-12-13T23:58:21.210 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112742.log.gz 2023-12-13T23:58:21.210 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100658.log 2023-12-13T23:58:21.211 INFO:teuthology.orchestra.run.smithi124.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.97326.log.gz 2023-12-13T23:58:21.211 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.104054.log.gz 2023-12-13T23:58:21.211 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98781.log 2023-12-13T23:58:21.211 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105483.log 2023-12-13T23:58:21.212 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.106075.log.gz/var/log/ceph/ceph-client.admin.100658.log: 2023-12-13T23:58:21.212 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102848.log 2023-12-13T23:58:21.212 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100658.log.gz 2023-12-13T23:58:21.212 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105483.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.98781.log.gz 2023-12-13T23:58:21.212 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107012.log 2023-12-13T23:58:21.213 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102848.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.105483.log.gz 2023-12-13T23:58:21.213 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100332.log 2023-12-13T23:58:21.213 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107012.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.102848.log.gz 2023-12-13T23:58:21.213 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101676.log 2023-12-13T23:58:21.213 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107012.log.gz 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102974.log 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101676.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.100332.log.gz 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106201.log 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102974.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101676.log.gz 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112784.log 2023-12-13T23:58:21.214 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106201.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.102974.log.gz 2023-12-13T23:58:21.215 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100800.log 2023-12-13T23:58:21.215 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106201.log.gz 2023-12-13T23:58:21.215 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100080.log 2023-12-13T23:58:21.215 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100800.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112784.log.gz 2023-12-13T23:58:21.216 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111706.log 2023-12-13T23:58:21.216 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97284.log 2023-12-13T23:58:21.216 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100800.log.gz 2023-12-13T23:58:21.216 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100080.log.gz 2023-12-13T23:58:21.216 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96696.log 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7%/var/log/ceph/ceph-client.admin.97284.log: -- replaced with /var/log/ceph/ceph-client.admin.111706.log.gz 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106301.log 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101088.log 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.97284.log.gz 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106301.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.96696.log.gz 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98613.log 2023-12-13T23:58:21.217 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106301.log.gz 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100926.log 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101088.log: /var/log/ceph/ceph-client.admin.98613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110032.log 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101088.log.gz 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98613.log.gz 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106343.log 2023-12-13T23:58:21.218 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100926.log.gz 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105119.log 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106343.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.110032.log.gz -5 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.106427.log 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106343.log.gz 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95562.log 2023-12-13T23:58:21.219 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105119.log.gz 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104238.log 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95562.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106427.log.gz 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100416.log 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95562.log.gz 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107851.log 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94868.log 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104238.log.gz 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107851.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.100416.log.gz 2023-12-13T23:58:21.220 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100164.log 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.107851.log.gz 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105219.log 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.94868.log.gz 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99776.log 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105219.log: gzip -5 87.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.100164.log.gz 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.96496.log 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105219.log.gz 2023-12-13T23:58:21.221 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99776.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.99776.log.gz 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.103467.log 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89139.log 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.103467.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96496.log.gz 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111832.log 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.89139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111580.log 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111832.log: gzip -5 --verbose 90.1% -- -- replaced with /var/log/ceph/ceph-client.admin.89139.log.gz /var/log/ceph/ceph-client.admin.104096.log 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111832.log.gz 2023-12-13T23:58:21.222 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101802.log 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.111580.log.gz 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103100.log 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.103467.log.gz 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107535.log 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101802.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.104096.log.gz 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101298.log 2023-12-13T23:58:21.223 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101802.log.gz 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107535.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.103100.log.gz 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103928.log 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107535.log.gz 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103970.log 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103928.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.101298.log.gz -5 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.112364.log 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103928.log.gz 2023-12-13T23:58:21.224 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99960.log 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112364.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103970.log.gz 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96156.log 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99960.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.112364.log.gz 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.97846.log 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99960.log.gz 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106970.log 2023-12-13T23:58:21.225 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.97846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108528.log 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96156.log.gz 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96498.log 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106970.log.gz 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96906.log 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.97846.log.gz 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96864.log 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96498.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108528.log.gz 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102890.log 2023-12-13T23:58:21.226 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.96498.log.gz 2023-12-13T23:58:21.227 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96906.log.gz 2023-12-13T23:58:21.227 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96497.log 2023-12-13T23:58:21.227 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.110821.log 2023-12-13T23:58:21.227 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.96864.log.gz 2023-12-13T23:58:21.227 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96497.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.102890.log.gz 2023-12-13T23:58:21.228 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96164.log 2023-12-13T23:58:21.228 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.110821.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.96497.log.gz 2023-12-13T23:58:21.228 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98949.log 2023-12-13T23:58:21.228 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107198.log 2023-12-13T23:58:21.229 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96164.log.gz 2023-12-13T23:58:21.229 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99650.log 2023-12-13T23:58:21.229 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108234.log 2023-12-13T23:58:21.229 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107198.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98949.log.gz 2023-12-13T23:58:21.230 INFO:teuthology.orchestra.run.smithi124.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.0.110821.log.gz 2023-12-13T23:58:21.230 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99650.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-13T23:58:21.230 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107198.log.gz 2023-12-13T23:58:21.230 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112000.log 2023-12-13T23:58:21.230 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108234.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99650.log.gz 2023-12-13T23:58:21.231 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101340.log 2023-12-13T23:58:21.231 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108234.log.gz 2023-12-13T23:58:21.231 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94742.log 2023-12-13T23:58:21.231 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101340.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112000.log.gz 2023-12-13T23:58:21.232 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97200.log 2023-12-13T23:58:21.232 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101340.log.gz 2023-12-13T23:58:21.232 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101928.log 2023-12-13T23:58:21.232 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97200.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.94742.log.gz 2023-12-13T23:58:21.233 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101634.log 2023-12-13T23:58:21.233 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-13T23:58:21.233 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97200.log.gz 2023-12-13T23:58:21.233 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111074.log 2023-12-13T23:58:21.234 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101928.log.gz 2023-12-13T23:58:21.234 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97967.log 2023-12-13T23:58:21.234 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111074.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101634.log.gz 2023-12-13T23:58:21.234 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99387.log 2023-12-13T23:58:21.235 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97967.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111074.log.gz 2023-12-13T23:58:21.235 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94657.log 2023-12-13T23:58:21.235 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99387.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97967.log.gz 2023-12-13T23:58:21.235 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108318.log 2023-12-13T23:58:21.236 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99387.log.gz 2023-12-13T23:58:21.236 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104364.log 2023-12-13T23:58:21.236 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95058.log 2023-12-13T23:58:21.236 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.94657.log.gz 2023-12-13T23:58:21.237 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104364.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.108318.log.gz 2023-12-13T23:58:21.237 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113094.log 2023-12-13T23:58:21.237 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.104364.log.gz 2023-12-13T23:58:21.237 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107935.log 2023-12-13T23:58:21.238 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.113094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109522.log 2023-12-13T23:58:21.238 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.95058.log.gz 2023-12-13T23:58:21.238 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106718.log 2023-12-13T23:58:21.239 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.113094.log.gz 2023-12-13T23:58:21.239 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107935.log.gz 2023-12-13T23:58:21.239 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108486.log 2023-12-13T23:58:21.239 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106718.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109522.log.gz 2023-12-13T23:58:21.240 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106844.log 2023-12-13T23:58:21.240 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3%/var/log/ceph/ceph-client.admin.108486.log: -- replaced with /var/log/ceph/ceph-client.admin.106718.log.gz 2023-12-13T23:58:21.240 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101256.log 2023-12-13T23:58:21.240 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106844.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108486.log.gz 2023-12-13T23:58:21.241 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94954.log 2023-12-13T23:58:21.241 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101256.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106844.log.gz 2023-12-13T23:58:21.241 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98655.log 2023-12-13T23:58:21.241 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101256.log.gz 2023-12-13T23:58:21.242 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96157.log 2023-12-13T23:58:21.242 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98655.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94954.log.gz 2023-12-13T23:58:21.242 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101214.log 2023-12-13T23:58:21.242 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98655.log.gz 2023-12-13T23:58:21.242 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111622.log 2023-12-13T23:58:21.243 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96157.log.gz/var/log/ceph/ceph-client.admin.101214.log: 2023-12-13T23:58:21.243 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108570.log 2023-12-13T23:58:21.243 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111622.log: 88.1%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.101214.log.gz --verbose 2023-12-13T23:58:21.243 INFO:teuthology.orchestra.run.smithi124.stderr: -- /var/log/ceph/ceph-client.admin.104951.log 2023-12-13T23:58:21.244 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111538.log 2023-12-13T23:58:21.244 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111622.log.gz 2023-12-13T23:58:21.244 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96161.log 2023-12-13T23:58:21.244 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104951.log.gz 2023-12-13T23:58:21.244 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109480.log 2023-12-13T23:58:21.245 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111538.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.108570.log.gz 2023-12-13T23:58:21.245 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108636.log 2023-12-13T23:58:21.245 INFO:teuthology.orchestra.run.smithi124.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.111538.log.gz 2023-12-13T23:58:21.245 INFO:teuthology.orchestra.run.smithi124.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.96161.log.gz 2023-12-13T23:58:21.246 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100248.log 2023-12-13T23:58:21.246 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.109480.log.gz/var/log/ceph/ceph-client.admin.108636.log: 2023-12-13T23:58:21.246 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103752.log 2023-12-13T23:58:21.246 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100248.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.108636.log.gz 2023-12-13T23:58:21.247 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112700.log 2023-12-13T23:58:21.247 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103752.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.100248.log.gz 2023-12-13T23:58:21.247 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.110316.log 2023-12-13T23:58:21.247 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.103752.log.gz 2023-12-13T23:58:21.248 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102204.log 2023-12-13T23:58:21.248 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112700.log.gz 2023-12-13T23:58:21.248 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96500.log 2023-12-13T23:58:21.248 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102204.log: gzip 89.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.110316.log.gz -- 2023-12-13T23:58:21.249 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.104196.log 2023-12-13T23:58:21.249 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102204.log.gz 2023-12-13T23:58:21.249 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106159.log 2023-12-13T23:58:21.249 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96500.log.gz 2023-12-13T23:58:21.250 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95823.log 2023-12-13T23:58:21.250 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89509.log 2023-12-13T23:58:21.250 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104196.log.gz 2023-12-13T23:58:21.250 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106159.log.gz 2023-12-13T23:58:21.250 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109254.log 2023-12-13T23:58:21.251 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95823.log.gz 2023-12-13T23:58:21.251 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.89509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101466.log 2023-12-13T23:58:21.251 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109254.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.89509.log.gz 2023-12-13T23:58:21.252 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103668.log 2023-12-13T23:58:21.252 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101466.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.109254.log.gz 2023-12-13T23:58:21.252 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104909.log 2023-12-13T23:58:21.252 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101466.log.gz 2023-12-13T23:58:21.253 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102626.log 2023-12-13T23:58:21.253 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104909.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103668.log.gzgzip 2023-12-13T23:58:21.253 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.110174.log 2023-12-13T23:58:21.253 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.104909.log.gz 2023-12-13T23:58:21.254 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102626.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-13T23:58:21.254 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110174.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.102626.log.gz 2023-12-13T23:58:21.254 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101004.log 2023-12-13T23:58:21.254 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-osd.1.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110174.log.gz 2023-12-13T23:58:21.255 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95226.log 2023-12-13T23:58:21.255 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89389.log 2023-12-13T23:58:21.255 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.101004.log.gz 2023-12-13T23:58:21.255 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109396.log 2023-12-13T23:58:21.256 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102720.log 2023-12-13T23:58:21.256 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0%/var/log/ceph/ceph-client.admin.89389.log: -- replaced with /var/log/ceph/ceph-client.admin.95226.log.gz 2023-12-13T23:58:21.256 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109396.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.89389.log.gz 2023-12-13T23:58:21.256 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107054.log 2023-12-13T23:58:21.257 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109396.log.gz 2023-12-13T23:58:21.257 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95352.log 2023-12-13T23:58:21.257 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102720.log: /var/log/ceph/ceph-client.admin.107054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106676.log 2023-12-13T23:58:21.257 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.102720.log.gz 2023-12-13T23:58:21.258 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107054.log.gz 2023-12-13T23:58:21.258 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99876.log 2023-12-13T23:58:21.258 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113136.log 2023-12-13T23:58:21.258 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106676.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95352.log.gz 2023-12-13T23:58:21.259 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99201.log 2023-12-13T23:58:21.259 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.106676.log.gz 2023-12-13T23:58:21.259 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99876.log.gz 2023-12-13T23:58:21.259 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.113136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96162.log 2023-12-13T23:58:21.260 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112238.log 2023-12-13T23:58:21.260 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113136.log.gz 2023-12-13T23:58:21.260 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96162.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.99201.log.gz 2023-12-13T23:58:21.260 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101382.log 2023-12-13T23:58:21.261 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.96162.log.gz 2023-12-13T23:58:21.261 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100758.log 2023-12-13T23:58:21.261 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112238.log: /var/log/ceph/ceph-client.admin.101382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106928.log 2023-12-13T23:58:21.261 INFO:teuthology.orchestra.run.smithi124.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112238.log.gz 2023-12-13T23:58:21.262 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100758.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101382.log.gz 2023-12-13T23:58:21.262 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111116.log 2023-12-13T23:58:21.262 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.100758.log.gz 2023-12-13T23:58:21.262 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100002.log 2023-12-13T23:58:21.263 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106928.log.gz 2023-12-13T23:58:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99692.log 2023-12-13T23:58:21.263 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111116.log: /var/log/ceph/ceph-client.admin.100002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106385.log 2023-12-13T23:58:21.263 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111116.log.gz 2023-12-13T23:58:21.264 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99692.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.100002.log.gz 2023-12-13T23:58:21.264 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105405.log 2023-12-13T23:58:21.264 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102372.log 2023-12-13T23:58:21.264 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106385.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99692.log.gz 2023-12-13T23:58:21.264 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105405.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.106385.log.gz 2023-12-13T23:58:21.265 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107493.log 2023-12-13T23:58:21.265 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105405.log.gz 2023-12-13T23:58:21.265 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111748.log 2023-12-13T23:58:21.265 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104993.log 2023-12-13T23:58:21.266 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102372.log.gz 2023-12-13T23:58:21.266 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111748.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107493.log.gz 2023-12-13T23:58:21.266 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95646.log 2023-12-13T23:58:21.266 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111748.log.gz 2023-12-13T23:58:21.267 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.104993.log: -- /var/log/ceph/ceph-client.admin.112952.log 2023-12-13T23:58:21.267 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110258.log 2023-12-13T23:58:21.267 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.104993.log.gz 2023-12-13T23:58:21.267 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112952.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.95646.log.gz 2023-12-13T23:58:21.268 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101508.log 2023-12-13T23:58:21.268 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110258.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.112952.log.gz 2023-12-13T23:58:21.268 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100700.log 2023-12-13T23:58:21.268 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110258.log.gz 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106117.log 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112154.log 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101508.log.gz 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106117.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100700.log.gz 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108192.log 2023-12-13T23:58:21.269 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108444.log 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112154.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106117.log.gz 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97074.log 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112154.log.gz 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.108192.log.gz 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95268.log 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108444.log: /var/log/ceph/ceph-client.admin.97074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112658.log 2023-12-13T23:58:21.270 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.108444.log.gz 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97074.log.gz 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95016.log 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100574.log 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112658.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.95268.log.gz 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95016.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.112658.log.gz 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107283.log 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95016.log.gz 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96499.log 2023-12-13T23:58:21.271 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100574.log: /var/log/ceph/ceph-client.admin.107283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103058.log 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.100574.log.gz 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96499.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107283.log.gz 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107451.log 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103058.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96499.log.gz 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103058.log.gz 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95827.log 2023-12-13T23:58:21.272 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107451.log.gz 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97116.log 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-osd.3.log: /var/log/ceph/ceph-client.admin.95827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101718.log 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97116.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95827.log.gz 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102456.log 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97116.log.gz 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96990.log 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.101718.log.gz 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108020.log 2023-12-13T23:58:21.273 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.98339.log 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102456.log.gz 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96990.log: /var/log/ceph/ceph-client.admin.108020.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96990.log.gz 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111790.log 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100532.log 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.2.98339.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108020.log.gz 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104322.log 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111790.log.gz 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105525.log 2023-12-13T23:58:21.274 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104322.log: 91.0% -- replaced with /var/log/ceph/ceph-client.2.98339.log.gz 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100532.log.gz 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99033.log 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.104322.log.gz 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105321.log 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105525.log.gz 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103710.log 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99033.log.gz 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108402.log 2023-12-13T23:58:21.275 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105321.log.gz 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98503.log 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108402.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103710.log.gz 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106760.log 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98503.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.108402.log.gz 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112196.log 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98503.log.gz 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107770.log 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106760.log.gz 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111496.log 2023-12-13T23:58:21.276 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107770.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.112196.log.gz 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100842.log 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.107770.log.gz 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99608.log 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100842.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.111496.log.gz 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112322.log 2023-12-13T23:58:21.277 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100842.log.gz 2023-12-13T23:58:21.280 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96495.log 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112322.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99608.log.gz 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98907.log 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.112322.log.gz 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102540.log 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96495.log.gz 2023-12-13T23:58:21.281 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98907.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95822.log 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:gzip 88.6% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.98907.log.gz -- 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.95824.log 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102540.log: /var/log/ceph/ceph-client.admin.95822.log: 89.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.102540.log.gz 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.83065.log 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110132.log 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95824.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95822.log.gz 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph.tmp-client.admin.83065.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.83065.log.gz 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106802.log 2023-12-13T23:58:21.282 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95824.log.gz 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106469.log 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110132.log: /var/log/ceph/ceph-client.admin.106802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103142.log 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110132.log.gz 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106469.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.106802.log.gz -5 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.102932.log 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106469.log.gz 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110400.log 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105035.log 2023-12-13T23:58:21.283 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102932.log: /var/log/ceph/ceph-client.admin.110400.log: gzip 91.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.103142.log.gz -- 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.105991.log 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102932.log.gz 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110400.log.gz 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.94480.log 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105035.log: /var/log/ceph/ceph-client.admin.105991.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105035.log.gz 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr: -- /var/log/ceph/ceph-client.admin.105177.log 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.105991.log.gz 2023-12-13T23:58:21.284 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101424.log 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.94480.log.gz 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.105784.log 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101424.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105177.log.gz 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112490.log 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111958.log 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.105784.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.101424.log.gz 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101550.log 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111958.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112490.log.gz 2023-12-13T23:58:21.285 INFO:teuthology.orchestra.run.smithi124.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107690.log 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101550.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111958.log.gz 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99734.log 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101550.log.gz 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107690.log: gzip -5 --verbose -- 91.4% /var/log/ceph/ceph-client.admin.94784.log 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.0.105784.log.gz 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99734.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.107690.log.gz 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112280.log 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99734.log.gz 2023-12-13T23:58:21.286 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102246.log 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94784.log: /var/log/ceph/ceph-client.admin.112280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.104701.log 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.94784.log.gz 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102246.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.112280.log.gz 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96380.log 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.102246.log.gz 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.104701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101760.log 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96380.log: /var/log/ceph/ceph-client.admin.101760.log: gzip -5 --verbose -- 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96380.log.gz 2023-12-13T23:58:21.287 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.94910.log 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-mgr.y.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.101760.log.gz 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103886.log 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5/var/log/ceph/ceph-client.admin.94910.log: --verbose 91.0% -- replaced with /var/log/ceph/ceph-client.0.104701.log.gz 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr: -- /var/log/ceph/ceph-client.admin.112532.log 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103886.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.94910.log.gz 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108108.log 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.103886.log.gz 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr:gzip/var/log/ceph/ceph-client.admin.112532.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.109354.log 2023-12-13T23:58:21.288 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112532.log.gz 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102804.log 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108108.log.gz 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102288.log 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr:gzip/var/log/ceph/ceph-client.admin.102804.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.102330.log 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109354.log.gz 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95310.log 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.102804.log.gz 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97428.log 2023-12-13T23:58:21.289 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.admin.102288.log.gz 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102330.log: /var/log/ceph/ceph-client.admin.95310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111454.log 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102330.log.gz 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95828.log 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.95310.log.gz 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97428.log: /var/log/ceph/ceph-client.admin.111454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108276.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97428.log.gz 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.111454.log.gz 2023-12-13T23:58:21.290 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101886.log 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95828.log.gz 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113054.log 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101886.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.108276.log.gz 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108064.log 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101886.log.gz 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.113054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101971.log 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107409.log 2023-12-13T23:58:21.291 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.113054.log.gz 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.108064.log: /var/log/ceph/ceph-client.admin.101971.log: gzip -5 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108064.log.gz 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.99261.log 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.101971.log.gz 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103828.log 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100290.log 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.107409.log.gz 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110358.log 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.99261.log.gz 2023-12-13T23:58:21.292 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103828.log.gz 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108360.log 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100290.log.gz 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102582.log 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110358.log: /var/log/ceph/ceph-client.admin.108360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96163.log 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110358.log.gz 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95142.log 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.108360.log.gz 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96163.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102582.log.gz 2023-12-13T23:58:21.293 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95821.log 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.96163.log.gz 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104154.log 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95142.log.gz 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110442.log 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95821.log: /var/log/ceph/ceph-client.admin.104154.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3%/var/log/ceph/ceph-client.admin.110442.log: -- replaced with /var/log/ceph/ceph-client.admin.95821.log.gz 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99918.log 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104154.log.gz 2023-12-13T23:58:21.294 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110442.log.gz 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107096.log 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105077.log 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99918.log: /var/log/ceph/ceph-client.admin.107096.log: gzip -5 --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99918.log.gz 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr: /var/log/ceph/ceph-client.admin.110484.log 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.107096.log.gz 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.105077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106259.log 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110484.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.105077.log.gz 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111412.log 2023-12-13T23:58:21.295 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110484.log.gz 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103624.log 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110562.log 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111412.log: -- replaced with /var/log/ceph/ceph-client.admin.106259.log.gz 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108150.log 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111412.log.gz 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94699.log 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.103624.log.gz 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110562.log.gz 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr: -5/var/log/ceph/ceph-client.admin.108150.log: --verbose -- /var/log/ceph/ceph-client.admin.95394.log 2023-12-13T23:58:21.296 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94699.log: gzip -5 --verbose 89.0% -- replaced with /var/log/ceph/ceph-client.admin.108150.log.gz 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr: -- /var/log/ceph/ceph-client.admin.99303.log 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95394.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.94699.log.gz 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95394.log.gz 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99471.log 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph.log: gzip 88.4% -5 --verbose -- /var/log/ceph/ceph-client.admin.97242.log 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.admin.99303.log.gz 2023-12-13T23:58:21.297 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99471.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4%/var/log/ceph/ceph-client.admin.97242.log: -- replaced with /var/log/ceph/ceph-client.admin.99471.log.gz 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr: 85.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106886.log 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97242.log.gz 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111200.log 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-osd.2.log: gzip/var/log/ceph/ceph-client.admin.106886.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.100616.log 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111200.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106886.log.gz 2023-12-13T23:58:21.298 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.100616.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111200.log.gz 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101844.log 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr: 88.6%/var/log/ceph/ceph-mds.c.log: -- replaced with /var/log/ceph/ceph-client.admin.100616.log.gz 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111916.log 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104442.log 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.101844.log.gz 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111916.log: 92.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112574.log 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111916.log.gz 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110995.log 2023-12-13T23:58:21.299 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.104442.log.gz 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107893.log 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.112574.log.gz/var/log/ceph/ceph-client.admin.110995.log: 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98991.log 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110995.log.gz 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107241.log 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107893.log: /var/log/ceph/ceph-client.admin.98991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107138.log 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.107893.log.gz 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107241.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98991.log.gz 2023-12-13T23:58:21.300 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98823.log 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.107241.log.gz 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97032.log 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94826.log 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107138.log.gz 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.97032.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.98823.log.gz 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107977.log 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99117.log 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97032.log.gz 2023-12-13T23:58:21.301 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107977.log: 89.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107367.log 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.admin.94826.log.gz 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107977.log.gz 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.99117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112616.log 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103016.log 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99117.log.gz 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107367.log.gz 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.112616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99513.log 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103016.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.112616.log.gz 2023-12-13T23:58:21.302 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112994.log 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.103016.log.gz/var/log/ceph/ceph-client.admin.99513.log: 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111242.log 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.99513.log.gz 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:gzip/var/log/ceph/ceph-client.admin.112994.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.106547.log 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.111242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101172.log 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112994.log.gz 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106547.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.111242.log.gz 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96155.log 2023-12-13T23:58:21.303 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.101172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107325.log 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106547.log.gz 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8%/var/log/ceph/ceph-client.admin.96155.log: -- replaced with /var/log/ceph/ceph-client.admin.101172.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96379.log 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102670.log 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96155.log.gz 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.96379.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.107325.log.gz 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104280.log 2023-12-13T23:58:21.304 INFO:teuthology.orchestra.run.smithi124.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.96379.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109170.log 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102670.log: /var/log/ceph/ceph-client.admin.104280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109296.log 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109170.log: 89.1% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.102670.log.gz 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.admin.104280.log.gz 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.109170.log.gz 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr: --verbose -- /var/log/ceph/ceph-client.admin.110090.log 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109212.log 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109296.log.gz 2023-12-13T23:58:21.305 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107577.log 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110090.log.gz 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.109212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112868.log 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.109212.log.gz 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94398.log 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.107577.log: /var/log/ceph/ceph-client.admin.112868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104012.log 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.107577.log.gz 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.94398.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.112868.log.gz 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-12-13T23:58:21.306 INFO:teuthology.orchestra.run.smithi124.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94398.log.gz 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.104012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98697.log 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103208.log 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104012.log.gz 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.98697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.108895.log 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.103208.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.98697.log.gz 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.103208.log.gz 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95826.log 2023-12-13T23:58:21.307 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.0.108895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106033.log 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.95826.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95826.log.gz 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102082.log 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.106033.log: 91.6% 88.9% -- replaced with /var/log/ceph/ceph-client.admin.106033.log.gz 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr: -- replaced with /var/log/ceph/ceph-client.0.108895.log.gz 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110216.log 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.102082.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.102082.log.gz 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr:/var/log/ceph/ceph-client.admin.110216.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.110216.log.gz 2023-12-13T23:58:21.308 INFO:teuthology.orchestra.run.smithi124.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-12-13T23:58:21.310 INFO:teuthology.orchestra.run.smithi124.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-13T23:58:21.310 INFO:teuthology.orchestra.run.smithi124.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-13T23:58:21.310 INFO:teuthology.orchestra.run.smithi124.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-13T23:58:21.310 INFO:teuthology.orchestra.run.smithi124.stderr: 93.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-13T23:58:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-13T23:58:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-13T23:58:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-13T23:58:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-13T23:58:21.311 INFO:teuthology.orchestra.run.smithi203.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-13T23:58:21.312 INFO:teuthology.orchestra.run.smithi203.stderr: 2023-12-13T23:58:21.312 INFO:teuthology.orchestra.run.smithi203.stderr:real 0m0.236s 2023-12-13T23:58:21.312 INFO:teuthology.orchestra.run.smithi203.stderr:user 0m1.085s 2023-12-13T23:58:21.312 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m0.062s 2023-12-13T23:58:21.331 INFO:teuthology.orchestra.run.smithi124.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-13T23:58:21.333 INFO:teuthology.orchestra.run.smithi124.stderr: 2023-12-13T23:58:21.333 INFO:teuthology.orchestra.run.smithi124.stderr:real 0m0.322s 2023-12-13T23:58:21.333 INFO:teuthology.orchestra.run.smithi124.stderr:user 0m1.146s 2023-12-13T23:58:21.333 INFO:teuthology.orchestra.run.smithi124.stderr:sys 0m0.237s 2023-12-13T23:58:21.334 INFO:tasks.ceph:Archiving logs... 2023-12-13T23:58:21.334 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi124/log 2023-12-13T23:58:21.335 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-13T23:58:21.812 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi203/log 2023-12-13T23:58:21.813 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-13T23:58:21.978 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-13T23:58:21.999 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-13T23:58:21.999 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-13T23:58:22.002 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-13T23:58:22.076 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-13T23:58:22.086 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-13T23:58:22.086 DEBUG:teuthology.orchestra.run.smithi124:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-13T23:58:22.089 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-13T23:58:22.102 INFO:teuthology.orchestra.run.smithi124.stderr:bash: line 1: ntpq: command not found 2023-12-13T23:58:22.102 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2023-12-13T23:58:22.104 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-13T23:58:22.104 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-12-13T23:58:22.104 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +114us[ +114us] +/- 93ms 2023-12-13T23:58:22.104 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -58us[ -58us] +/- 77ms 2023-12-13T23:58:22.104 INFO:teuthology.orchestra.run.smithi124.stdout:^* hv03.front.sepia.ceph.com 3 6 377 16 -29us[ -20us] +/- 75ms 2023-12-13T23:58:22.105 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 14 +125us[ +125us] +/- 93ms 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 14 -52us[ -52us] +/- 77ms 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv03.front.sepia.ceph.com 3 6 377 15 -21us[ -14us] +/- 75ms 2023-12-13T23:58:22.107 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-13T23:58:22.108 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-13T23:58:22.117 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-13T23:58:22.118 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-13T23:58:22.205 DEBUG:teuthology.orchestra.run.smithi124:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-13T23:58:22.309 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-13T23:58:22.404 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-13T23:58:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:58:22.437 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-13T23:58:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:58:22.469 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-13T23:58:22.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-13T23:58:22.518 INFO:teuthology.task.internal:Duration was 683.430637 seconds 2023-12-13T23:58:22.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-13T23:58:22.528 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-13T23:58:22.528 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-13T23:58:22.531 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-13T23:58:22.571 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-13T23:58:22.574 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-13T23:58:22.932 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-13T23:58:22.933 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi124.front.sepia.ceph.com 2023-12-13T23:58:22.933 DEBUG:teuthology.orchestra.run.smithi124:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-13T23:58:22.992 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2023-12-13T23:58:22.992 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-13T23:58:23.027 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-13T23:58:23.027 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-13T23:58:23.035 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-13T23:58:23.096 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-13T23:58:23.097 DEBUG:teuthology.orchestra.run.smithi124:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-13T23:58:23.103 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-13T23:58:23.232 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-13T23:58:23.243 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-13T23:58:23.243 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-13T23:58:23.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-13T23:58:23.297 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-13T23:58:23.328 DEBUG:teuthology.orchestra.run.smithi124:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-13T23:58:23.330 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-13T23:58:23.353 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = core 2023-12-13T23:58:23.362 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2023-12-13T23:58:23.378 DEBUG:teuthology.orchestra.run.smithi124:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-13T23:58:23.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:58:23.424 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-13T23:58:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-13T23:58:23.438 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-13T23:58:23.448 INFO:teuthology.task.internal:Transferring archived files... 2023-12-13T23:58:23.449 DEBUG:teuthology.misc:Transferring archived files from smithi124:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi124 2023-12-13T23:58:23.450 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-13T23:58:23.501 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669/remote/smithi203 2023-12-13T23:58:23.501 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-13T23:58:23.539 INFO:teuthology.task.internal:Removing archive directory... 2023-12-13T23:58:23.540 DEBUG:teuthology.orchestra.run.smithi124:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-13T23:58:23.542 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-13T23:58:23.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-13T23:58:23.599 INFO:teuthology.task.internal:Not uploading archives. 2023-12-13T23:58:23.599 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-13T23:58:23.638 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-13T23:58:23.638 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-13T23:58:23.641 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-13T23:58:23.652 INFO:teuthology.orchestra.run.smithi124.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 13 23:58 /home/ubuntu/cephtest 2023-12-13T23:58:23.655 INFO:teuthology.orchestra.run.smithi203.stdout: 265532 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 13 23:58 /home/ubuntu/cephtest 2023-12-13T23:58:23.656 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-13T23:58:23.670 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-13T23:58:23.789 INFO:teuthology.nuke:Checking targets against current locks 2023-12-13T23:58:23.822 DEBUG:teuthology.nuke:shortname: smithi124 2023-12-13T23:58:23.822 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-13T23:58:23.852 DEBUG:teuthology.nuke:shortname: smithi203 2023-12-13T23:58:23.853 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-13T23:58:23.856 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-13 23:38:45.291895', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPBqrEjhIl+df54aDgFucXunpO4UgObFWJ9ytkz5DhJ26N/0p3E8peMvYr//WC8rqdADbgVMPgTdeyRMyUl8nI8='} 2023-12-13T23:58:23.881 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-13_20:38:28-fs-wip-yuri10-testing-2023-12-12-1229-distro-default-smithi/7490669', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-13 23:38:45.293484', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJA14Vswc23fItM+D5R1LMCctN4imLdwSpzC1urUzN+ZX/tklb8W2t0fbPe7MwqXCylSmjMNwo57jeZjMMTgPO0='} 2023-12-13T23:58:23.902 INFO:teuthology.orchestra.console.smithi124:Power off 2023-12-13T23:58:23.903 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-13T23:58:23.929 DEBUG:teuthology.orchestra.console.smithi124:power off output: Chassis Power Control: Down/Off 2023-12-13T23:58:23.930 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:23.948 INFO:teuthology.orchestra.console.smithi203:Power off 2023-12-13T23:58:23.949 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-13T23:58:23.959 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-12-13T23:58:23.972 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2023-12-13T23:58:23.972 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:23.995 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2023-12-13T23:58:27.960 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:27.996 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:28.071 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-12-13T23:58:28.107 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2023-12-13T23:58:32.073 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:32.108 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-13T23:58:32.184 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is off 2023-12-13T23:58:32.219 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2023-12-13T23:58:32.286 INFO:teuthology.orchestra.console.smithi124:Power off completed 2023-12-13T23:58:32.321 INFO:teuthology.orchestra.console.smithi203:Power off completed 2023-12-13T23:58:32.425 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/pool-perm} duration: 683.4306371212006 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=18d2041c5ebc45a3905a4b3df3c8dfa5 status: fail success: false 2023-12-13T23:58:32.426 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-13T23:58:32.498 INFO:teuthology.run:FAIL